Software initialization of usb devices on a single bus

ABSTRACT

A method of initializing programmable devices on a shared bus, comprises, on power up, loading control instructions on a processor from an attached tangible, non-transient computer-readable medium, automatically allowing a first programmable device on a common bus to exit its reset state and enter a read-to-program state as the computer system powers up, automatically holding a second programmable device on the common bus in its reset state, querying the common bus by the processor to identify the first programmable device, copying the contents of a program image file by the processor from the computer-readable medium across the common bus to the first programmable device, and once the first programmable device has been programmed with the contents of the program image file, signaling the first programmable device to release the reset hold on the second programmable device.

TECHNICAL FIELD

The present disclosure relates to systems and methods for testingcommunications networks, services, and devices, e.g., testing thetraffic-handling performance and/or security of the network, networkaccessible devices, cloud services, and data center services.

BACKGROUND

Organizations are increasingly reliant upon the performance, security,and availability of networked applications to achieve business goals. Atthe same time, the growing popularity of latency-sensitive,bandwidth-heavy applications is placing heavy demands on networkinfrastructures. Further, cyber attackers are constantly evolving theirmode of assault as they target sensitive data, financial assets, andoperations. Faced with these performance demands and increasinglysophisticated security threats, network equipment providers (NEPs) andtelecommunications service providers (SPs) have delivered a newgeneration of high-performance, content-aware network equipment andservices.

Content-aware devices that leverage deep packet inspection (DPI)functionality have been around for several years, and new content-awareperformance equipment is coming to market each year. However, recenthigh-profile performance and security failures have brought renewedfocus to the importance of sufficient testing to ensure content-awarenetwork devices can perform under real-world and peak conditions. Thetraditional approach of simply reacting to attacks and traffic evolutionhas cost organizations and governments billions. Today's sophisticatedand complex high-performance network devices and the network they run onrequire a more comprehensive approach to testing prior to deploymentthan traditional testing tools are able to provide. NEPs, SPs, and otherorganizations require testing solutions capable of rigorously testing,simulating, and emulating realistic application workloads and securityattacks at line speed. Equally important, these testing tools must beable to keep pace with emerging and more innovative products as well asthoroughly vet complex content-aware/DPI-capable functionality byemulating a myriad of application protocols and other types of contentat ever-increasing speeds and feeds to ensure delivery of an outstandingquality of experience (QoE) for the customer and/or subscriber.

Network infrastructures today are typically built on IP foundations.However, measuring and managing application performance in relation tonetwork devices remain challenges. To make matters worse, content-awarenetworking mandates controls for Layers 4-7 as well as the traditionalLayer 2-3 attributes. Yet, to date, the bulk of the IP network testingindustry has focused primarily on testing of Layers 2-3 with minimalconsideration for Layers 4-7. Now with the rise of content-drivenservices, Layers 4-7 are increasingly strategic areas for networkoptimization and bulletproofing.

Even as NEPs and SPs rush to introduce newer, more sophisticatedcontent-aware/DPI-capable devices to reap the associated business andrecreational benefits these products deliver, the testing of thesedevices has remained stagnant. Legacy testing solutions and traditionaltesting practices typically focus on the IP network connection,especially routers and switches, and do not have sufficientfunctionality or capability to properly test this new class of devices.Nor are they aligned with content-driven approaches such as using andapplying test criteria using stateful blended traffic and live securitystrikes at line speeds. The introduction of content-aware functionalityinto the network drives many new variables for testing that resistcorner-case approaches and instead require realistic, randomized traffictesting at real-time speeds. The inability to test this new set ofcontent-aware and software-driven packet inspection devices contributesto the deployment challenges and potential failure of many of them oncethey are deployed.

SUMMARY OF THE INVENTION

In one embodiment, a method of initializing programmable devices on ashared bus, comprises, on power up, loading control instructions on aprocessor from an attached tangible, non-transient computer-readablemedium, automatically allowing a first programmable device on a commonbus to exit its reset state and enter a read-to-program state as thecomputer system powers up, automatically holding a second programmabledevice on the common bus in its reset state, querying the common bus bythe processor to identify the first programmable device, copying thecontents of a program image file by the processor from thecomputer-readable medium across the common bus to the first programmabledevice, and once the first programmable device has been programmed withthe contents of the program image file, signaling the first programmabledevice to release the reset hold on the second programmable device.

In another embodiment, a tangible, non-transitory computer-readablemedia comprises instructions that, when executed on a processor, enablethe processor to perform the steps comprising query a common bus by aprocessor to identify a first programmable device connected to thecommon bus, copy the contents of a program image file by the processorfrom the computer-readable medium across the common bus to the firstprogrammable device, and once the first programmable device has beenprogrammed with the contents of the program image file, signal the firstprogrammable device to release a reset hold on a second programmabledevice.

In yet another embodiment, a computing system comprises a processorattached to a tangible, non-transient computer-readable mediumcontaining instructions and a first and a second program image file, anda common bus connected to the processor and a first and a secondprogrammable device. In this embodiment, the first programmable deviceis configured to exit its reset mode automatically on power up, thesecond programmable device is configured to remain in its reset modeuntil signaled by the first programmable device, and the instructions,when executed by the processor, instruct the processor to perform thesteps comprising query the common bus to identify the first programmabledevice, copy the contents of a program image file by the processor fromthe computer-readable medium across the common bus to the firstprogrammable device, and once the first programmable device has beenprogrammed with the contents of the program image file, signal the firstprogrammable device to release a reset hold on a second programmabledevice.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete understanding of the present embodiments and advantagesthereof may be acquired by referring to the following description takenin conjunction with the accompanying drawings, in which like referencenumbers indicate like features, and wherein:

FIG. 1 illustrates a block diagram of an arrangement for testing theperformance of a communications network and/or one or more networkdevices using a network testing system according to certain embodimentsof the present disclosure;

FIGS. 2A-2G illustrate example topologies or arrangements in which anetwork testing system according to certain embodiments may be connectedto a test system, e.g., depending on the type of the test system and/orthe type of testing or simulation to be performed by the network testingsystem;

FIG. 3 illustrates an example configuration of a network testing system,according to an example embodiment;

FIG. 4 is a high-level illustration of an example architecture of a cardor blade of a network testing system, according to an exampleembodiment;

FIG. 5 is a more detailed illustration of the example testing andsimulation architecture shown in FIG. 4, according to an exampleembodiment;

FIGS. 6A and 6B illustrates relevant components and an example processflow, respectively, of an example high-speed, high-resolution networkpacket capture subsystem of a network testing system, according to anexample embodiment;

FIGS. 7A and 7B illustrates relevant components and an example processflow, respectively, of an example high-speed packet generation andmeasurement subsystem of a network testing system, according to anexample embodiment;

FIGS. 8A and 8B illustrates relevant components and an example processflow, respectively, of an example application-level simulation andmeasurement subsystem of a network testing system, according to anexample embodiment;

FIGS. 9A and 9B illustrates relevant components and an example processflow, respectively, of an example security and exploit simulation andanalysis subsystem of a network testing system, according to an exampleembodiment;

FIG. 10 illustrates relevant components of an example statisticscollection and reporting subsystem of a network testing system,according to an example embodiment;

FIG. 11 illustrates a layer-based view of an example application systemarchitecture of a network testing system, according to exampleembodiments;

FIG. 12 illustrates select functional capabilities implemented by of anetwork testing system, according to certain embodiments;

FIG. 13A illustrates example user application level interfaces to anetwork testing system, according to example embodiments;

FIG. 13B illustrates example user application level interfaces to anetwork testing system, according to example embodiments;

FIG. 13C illustrates an example user interface screen for configuringaspects of a network testing system, according to an example embodiment;

FIG. 13D illustrates an example interface screen for configuring anetwork testing application, according to an example embodiment;

FIGS. 14A-14B illustrate a specific implementation of the architectureof a network testing system, according to one example embodiment;

FIG. 15 illustrates an example of an alternative architecture of thenetwork testing system, according to an example embodiment;

FIG. 16 illustrates various sub-systems configured to provide variousfunctions associated with a network testing system, according to anexample embodiment;

FIG. 17 illustrates an example layout of Ethernet packets containing CLDcontrol messages for use in a network testing system, according tocertain embodiments;

FIG. 18 illustrates an example register access directive for writingdata to CLD registers in a network testing system, according to certainembodiments;

FIG. 19 illustrates an example flow of the life of a register accessdirective in a network testing system, according to an exampleembodiment;

FIG. 20 illustrates an example DHCP-based boot management system in anetwork testing system, according to an example embodiment;

FIG. 21 illustrates an example DHCP-based boot process for a card orblade of a network testing system, according to an example embodiment;

FIG. 22 illustrates an example method for generating a configurationfile during a DHCP-based boot process in a network testing system,according to an example embodiment;

FIG. 23 illustrates portions of an example packet processing and routingsystem of a network testing system, according to an example embodiment;

FIG. 24 illustrates an example method for processing and routing a datapacket received by a network testing system using the example packetprocessing and routing system of FIG. 23, according to an exampleembodiment;

FIG. 25 illustrates a process of dynamic routing determination in anetwork testing system, according to an example embodiment;

FIG. 26 illustrates an efficient packet capture memory system for anetwork testing system, according to an example embodiment;

FIG. 27 illustrates two example methods for capturing network data in anetwork testing system, according to an example embodiment;

FIG. 28 illustrates two data loopback scenarios that may be supported bya network testing system, according to an example embodiment;

FIG. 29 illustrates two example arrangements for data loopback andpacket capture in a capture buffer of a network testing system,according to example embodiments;

FIG. 30 illustrates aspects an example loopback and capture system in anetwork testing system, according to an example embodiment;

FIG. 31 illustrates example routing and/or capture of data packets in avirtual wire internal loopback scenario and an external loopbackscenario provided in a network testing system, according to an exampleembodiment;

FIG. 32 illustrates an example multiple-domain hash table for use in anetwork testing system, according to an example embodiment;

FIG. 33 illustrates an example process for looking up a linked listelement based on a first key value, according to an example embodiments;

FIG. 34 illustrates an example process for looking up a linked listelement 686 based on a second key value, according to an exampleembodiments;

FIG. 35 illustrates an example segmentation offload process in a networktesting system, according to an example embodiment;

FIG. 36 illustrates another example segmentation offload process in anetwork testing system, according to an example embodiment;

FIG. 37 illustrates an example packet assembly system of a networktesting system, according to an example embodiment;

FIG. 38 illustrates an example process performed by a receive statemachine (Rx) TCP segment assembly offload, according to an exampleembodiment;

FIG. 39 illustrates an example process performed by s transmit statemachine (Tx) for TCP segment assembly offload, according to an exampleembodiment;

FIG. 40 illustrates an example method for allocating resources ofnetwork processors in a network testing system, according to an exampleembodiment;

FIGS. 41A-41E illustrate a process flow of an algorithm for determiningwhether a new test can be added to a set of tests running on a networktesting system, and if so, distributing the new test to one or morenetwork processors of the network testing system, according to anexample embodiment;

FIG. 42 illustrates an example method for implementing the algorithm ofFIGS. 41A-41E in a network testing system, according to an exampleembodiment;

FIG. 43 illustrates the latency performance of an example device orinfrastructure under test by a network testing system, as presented to auser, according to an example embodiment;

FIG. 44 is an example table of a subset of the raw statistical data fromwhich the chart of FIG. 43 may be derived, according to an exampleembodiment;

FIG. 45 is an example method for determining dynamic latency bucketsaccording to an example embodiment of the present disclosure;

FIG. 46 illustrates an example serial port access system in a networktesting system, according to an example embodiment;

FIG. 47 illustrates an example method for setting up an intra-bladeserial connection in a network testing system, e.g., when a processorneeds to connect to a serial port on the same blade, according to anexample embodiment;

FIG. 48 illustrates an example method for setting up an inter-bladeconnection between a requesting device on a first blade with a targetdevice on a second blade in a network testing system, according to anexample embodiment;

FIG. 49 illustrates an example USB device initiation system for use in anetwork testing system, according to an example embodiment;

FIG. 50 illustrates an example method for managing the discovery andinitiation of microcontrollers in the USB device initiation system ofFIG. 49, according to an example embodiment;

FIG. 51 illustrates an example serial bus based CLD programming systemin a network testing system, according to an example embodiment;

FIG. 52 illustrates an example programming process implemented by theserial bus based CLD programming system of FIG. 51, according to anexample embodiment;

FIG. 53 illustrates an example JTAG-based debug system of a networktesting system, according to an example embodiment;

FIG. 54 illustrates a three-dimensional view of an example networktesting system having three blades installed in a chassis, according toan example embodiment;

FIGS. 55A-59B illustrate various views of an example arrangement ofdevices on a card of a network testing system, at various stages ofassembly, according to an example embodiment;

FIG. 60 shows a three-dimensional isometric view of an example dual-bodyheat sink for use in a network testing system, according to an exampleembodiment;

FIG. 61 shows a top view of the dual-body heat sink of FIG. 60,according to an example embodiment;

FIG. 62 shows a bottom view of the dual-body heat sink of FIG. 60,according to an example embodiment;

FIG. 63 shows a three-dimensional isometric view from above of anexample air baffle for use in heat dissipation system of a networktesting system, according to an example embodiment;

FIGS. 64A and 64B shows a three-dimensional exploded view from below,and a three-dimensional assembled view from below, of the air baffle ofFIG. 63, according to an example embodiment;

FIG. 65 shows a side view of the assembled air baffle of FIG. 63,illustrating air flow paths promoted by the air baffle, according to anexample embodiment;

FIG. 66 illustrates an assembled drive carrier of a drive assembly ofnetwork testing system, according to an example embodiment;

FIG. 67 shows an exploded view of the drive carrier of FIG. 68,according to an example embodiment;

FIGS. 68A and 68B shows three-dimensional isometric views of a drivecarrier support for receiving the drive carrier of FIG. 68, according toan example embodiment;

FIG. 69 illustrates a drive branding solution, according to certainembodiments of the present disclosure; and

FIG. 70 illustrates branding and verification processes, according tocertain embodiments of the present disclosure.

DETAILED DESCRIPTION

Preferred embodiments and their advantages over the prior art are bestunderstood by reference to FIGS. 1-70 below in view of the followinggeneral discussion.

FIG. 1 illustrates a general block diagram of an arrangement 10 fortesting the performance of a communications network 12 and/or one ormore network devices 14 using a network testing system 16, according tocertain embodiments of the present disclosure. Test devices 14 may bepart of a network 12 tested by network testing system 16, or may beconnected to network testing system 16 by network 12. Thus, networktesting system 16 may be configured for testing network 12 and/ordevices 14 within or connected to network 12. For the sake ofsimplicity, the test network 12 and/or devices 14 are referred to hereinas the test system 18. Thus, a test system 18 may comprise a network 12,one or more devices 14 within a network 12 or coupled to a network 12,one or more hardware, software, and/or firmware components of device(s)14, or any other component or aspect of a network or network device.

Network testing system 16 may be configured to test the performance(e.g., traffic-handling performance) of devices 14, the security of atest system 18 (e.g., from security attacks), or both the performanceand security of a test system 18. In some embodiments, network testingsystem 16 configured to simulate a realistic combination of business,recreational, malicious, and proprietary application traffic atsufficient speeds to test both performance and security together usingthe same data and tests. In some embodiments, network testing system 16is configured for testing content-aware systems 18 devices 14 and/orcontent-unaware systems 18.

Network 12 may include any one or more networks which may be implementedas, or may be a part of, a storage area network (SAN), personal areanetwork (PAN), local area network (LAN), a metropolitan area network(MAN), a wide area network (WAN), a wireless local area network (WLAN),a virtual private network (VPN), an intranet, the Internet or any otherappropriate architecture or system that facilitates the communication ofsignals, data and/or messages (generally referred to as data) via anyone or more wired and/or wireless communication links.

Devices 14 may include any type or types of network device, e.g.,servers, routers, switches, gateways, firewalls, bridges, hubs,databases or data centers, workstations, desktop computers, wirelessaccess points, wireless access devices, and/or any other type or typesof devices configured to communicate with other network devices over acommunications medium. Devices 14 may also include any hardware,software, and/or firmware components of any such network device, e.g.,operating systems, applications, CPUs, configurable logic devices(CLDs), application-specific integrated circuits (ASICs), etc.

In some embodiments, network testing system 16 is configured to modeland simulate network traffic. The network testing system 16 may act asvirtual infrastructure and simulate traffic behavior of network devices(e.g., database server, Web server) running a specific application. Theresulting network traffic originated from the network testing system 16may drive the operation of a test system 18 for evaluating theperformance and/or security of the system 18. Complex models can bebuilt on realistic applications such that a system 18 can be tested andevaluated under realistic conditions, but in a testing environment.Simultaneously, network testing system 16 may monitor the performanceand/or security of a test system 18 and may collect various metrics thatmeasure performance and/or security characteristics of system 18.

In some embodiments, network testing system 16 comprises a hardware- andsoftware-based testing and simulation platform that includes of a numberof interconnected subsystems. These systems may be configured to operateindependently or in concert to provide a full-spectrum solution fortesting and verifying network performance, application and securitytraffic scenarios. These subsystems may be interconnected in a manner toprovide high-performance, highly-accurate measurements and deepintegration of functionality.

For example, as shown in FIG. 1, network testing system 16 may compriseany or all of the following testing and simulation subsystems: ahigh-speed, high-resolution network packet capture subsystem 20, ahigh-speed packet generation and measurement subsystem 22, anapplication-level simulation and measurement subsystem 24, a securityand exploit simulation and analysis subsystem 26, and/or a statisticscollection and reporting subsystem 28. Subsystems 20-28 are discussedbelow in greater detail. In some embodiments, the architecture ofnetwork testing system 16 may allow for some or all of subsystems 20-28to operate simultaneously and cooperatively within the same software andhardware platform. Thus, in some embodiments, system 16 is configured togenerate and analyze packets at line rate, while simultaneouslycapturing that same traffic, performing application simulation, andsecurity testing. In particular embodiments, system 16 comprises customhardware and software arranged and programmed to deliver performance andmeasurement abilities not achievable with conventional software orhardware solutions.

Network testing system 16 may be connected to the test system 18 in anysuitable manner, e.g., according to any suitable topology orarrangement. In some embodiments or arrangements, network testing system16 may be connected on both sides of a system 18 to be tested, e.g., tosimulate both clients and servers passing traffic through the testsystem. In other embodiment or arrangements, network testing system 16may be connected to any entry point to the test system 18, e.g., to actas a client to the test system 18. In some embodiment or arrangements,network testing system 16 may act in both of these modes simultaneously.

FIGS. 2A-2G illustrate example topologies or arrangements in whichnetwork testing system 16 may be connected to a test system 18, e.g.,depending on the type of the test system 18 and/or the type of testingor simulation to be performed by network testing system 16.

FIG. 2A illustrates an example arrangement for testing a data center 18using network testing system 16, according to an example embodiment. Adata center 18 may include a collection of virtual machines (VMs), eachspecialized to run one service per VM, wherein the number of VMsdedicated to each service may be configurable. For example, as shown,data center 18 may include the following VMs: a file server 14 a, a webserver 14 b, a mail server 14 c, and a database server 14 d, which maybe integrated in the same physical device or devices, or communicativelycoupled to each other via a network 12, which may comprise one or morerouters, switches, and/or other communications links. In this examplearrangement, network testing system 16 is connected to data center 18 bya single interface 40. Network testing system 16 may be configured toevaluate the data center 18 based on (a) its performance and resiliencyin passing specified traffic. In other embodiments, network testingsystem 16 may be configured to evaluate the ability of the data center18 to block malicious traffic.

FIG. 2B illustrates an example arrangement for testing a firewall 18using network testing system 16, according to an example embodiment.Firewall 18 may comprise, for example, a device which connects multiplelayer 3 networks and applies a security polity to traffic passingthrough. Network testing system 16 may be configured to test thefirewall 18 based on its performance and resiliency in passingspecifically allowed traffic and its ability to withstand packet andprotocol corruption. In this example arrangement, network testing system16 is connected to firewall 18 by two interface 40 a and 40 b, e.g.,configured to use Network Address Translation (NAT).

FIGS. 2C-2E illustrate example arrangements for testing an LTE networkusing network testing system 16, according to an example embodiment. Asshown in FIGS. 2C-2E, an LTE network may comprise the SystemArchitecture Evolution (SAE) network architecture of the 3GPP LTEwireless communication standard. According to the SAE architecture, userequipment (UEs) may be wirelessly connected to a mobility managemententity (MME) and/or serving gateway (SGW) via eNodeB interface. A homesubscriber server (HSS) may be connected to the MME, and the SGW may beconnected to a packet data network gateway (PGW), configured forconnecting network 18 to a public data network 42, e.g., the Internet.

In some embodiment, network testing system 16 may be configured tosimulated various components of an LTE network in order to test othercomponents or communication links of the LTE network 18. FIGS. 2C-2Eillustrate three example arrangements in which system 16 simulatesdifferent portions or components of the LTE network in order to testother components or communication links of the LTE network (i.e., thetested system 18). In each figure, the portions or components 18 of theLTE network that are simulated by system 16 are indicated by adouble-line outline, and connections between network testing system 16and the tested components 18 of the LTE network are indicated by dashedlines and reference number 40.

In the example arrangement shown in FIG. 2C, network testing system 16may be configured to simulate user equipment (UEs) and eNodeB interfacesat one end of the LTE network, and a public data network 42 (e.g.,Internet devices) connected to the other end of the LTE network. Asshown, network testing system 16 may be connected to the tested portion18 of the LTE network by connections 40 that simulate the following LTEnetwork connections: (a) S1-MME connections between eNodeB interfacesand the MME, (b) S1-U connection between eNodeB interfaces and the SGW;and (c) SGi connection between the PGW and public data network 42 (e.g.,Internet devices).

The example arrangement shown in FIG. 2D is largely similar to theexample arrangement of FIG. 2C, but the MME is also simulated by networktesting system 16, and the LTE network is connected to an actual publicdata network 42 (e.g., real Internet servers) rather than simulating thepublic data network 42 using system 16. Thus, as shown, network testingsystem 16 is connected to the tested portion 18 of the LTE network byconnections 40 that simulate the following LTE network connections: (a)S1-U connection between eNodeB interfaces and the SGW, and (b) S11connection between the MME and SGW.

In the example arrangement shown in FIG. 2E, network testing system 16is configured to simulate all components of the LTE network, with theexpectation that a deep packet inspection (DPI) device, e.g., afirewall, intrusion detection or prevention device (e.g., IPS or IDS),load balancer, etc., will be watching and analyzing the traffic oninterfaces S1-U and S11. Thus, network testing system 16 may test theperformance of the DPI device.

FIG. 2F illustrates an example arrangement for testing an applicationserver 18 using network testing system 16, according to an exampleembodiment. Application server 18 may comprise, for example, a virtualmachine (VM) with multiple available services (e.g., mail, Web, SQL, andfile sharing). Network testing system 16 may be configured to evaluatethe application server 18 based on its performance and resiliency inpassing specified traffic. In this example arrangement, network testingsystem 16 is connected to application server 18 by one interface 40.

FIG. 2G illustrates an example arrangement for testing a switch 18 usingnetwork testing system 16, according to an example embodiment. Switch 18may comprise, for example, a layer 2 networking device that connectsdifferent segments on the same layer 3 network. Network testing system16 may be configured to test the switch 18 based on its performance andresiliency against frame corruption. In this example arrangement,network testing system 16 is connected to switch 18 by two interface 40a and 40 b.

FIG. 3 illustrates an example configuration of a network testing system16, according to example embodiments. Network testing system 16 mayinclude a chassis 50 including any suitable number of slots 52, eachconfigured to receive a modular card, or blade, 54. A card or blade 54may comprise one or more printed circuit boards (e.g., PCB 380 discussedbelow). For example, as shown, chassis 50 may include Slot 0 configuredto receive Card 0, Slot 1 configured to receive Card 1, . . . and Slot nconfigured to receive Card n, where n equals any suitable number, e.g.,1, 2, 3, 4, 5, 7, or more. For example, in some embodiments, chassis 50is a 3-slot chassis, a 5-slot chassis, or a 12-slot chassis. In otherembodiments, system 16 comprises a single card 54.

Each card 54 may be plugged into a backplane 56, which may includephysical connections 60 for communicatively connecting cards 54 to eachother, as discussed below. While cards may be interconnected, each cardis treated for some purposes as an independent unit. Communicationswithin a card are considered to be “local” communications. Two differentcards attached to the same backplane may be running different versionsof software so long as the versions are compatible.

Each card 54 may include any architecture 100 of hardware, software,and/or firmware components for providing the functionality of networktesting system 16. For example, card 0 may include an architecture 100a, card 1 may include an architecture 100 b, . . . , and card n mayinclude an architecture 100 n. The architecture 100 of each card 54 maybe the same as or different than the architecture 100 of each other card54, e.g., in terms of hardware, software, and/or firmware, andarrangement thereof.

Each architecture 100 may include a system controller, one or morenetwork processors, and one or more CLDs connected to a managementswitch 110 (and any other suitable components, e.g., memory devices,communication interfaces, etc.). Cards 54 may be communicatively coupledto each other via the backplane 56 and management switches 110 of therespective cards 54, as shown in FIG. 3. In some embodiments, backplane56 include physical connections for connecting each card 54 directly toeach other card 54. Thus, each card 54 may communicate with each othercard 54 via the management switches 110 of the respective cards 54,regardless of whether one or more slots 52 are empty or whether one ormore cards 54 are removed.

In some embodiments, each card 54 may be configured to operate byitself, or cooperatively with one or more other cards 54, to provide anyof the functionality discussed herein.

FIG. 4 is an high-level illustration of an example architecture 100A ofa card 54 of network testing system 16, according to an exampleembodiment. As shown, example architecture 100A, referred to as a“testing and simulation architecture,” may include a controller 106, twonetwork processors 105 and multiple CLDs 102 coupled to a managementswitch 110, and memory 103 coupled to the CLDs 102.

In general, controller 106 is programmed to initiate and coordinate manyof the functions of network testing system 16. In some embodiments,controller 106 may be a general purpose central processing unit (CPU)such as an Intel x86 compatible part. Controller 106 may run ageneral-purpose multitasking or multiprocessing operating system such asa UNIX or Linux variant.

In general, network processors 105 are programmed to generate outboundnetwork data in the form of one or more data packets and are programmedto receive and process inbound network data in the form of one or moredata packets. In some embodiments, network processors 105 may be generalpurpose CPUs. In other embodiments, network processors 105 may bespecialized CPUs with instruction sets and hardware optimized forprocessing network data. For example, network processors may be selectedfrom the Netlogic XLR family of processors.

Configurable logic devices (CLDs) 102 provide high-performance,specialized computation, data transfer, and data analysis capabilitiesto process certain data or computation intensive tasks at or near thenetwork line rates.

As used herein, the term configurable logic device (CLD) means a devicethat includes a set of programmable logic units, internal memory, andhigh-speed internal and external interconnections. Examples of CLDsinclude field programmable gate arrays (FPGAs) (e.g., ALTERA STRATIXfamily, XILINX VIRTEX family, as examples), programmable logic devices(PLDs), programmable array logic devices (PAL), and configurableprogrammable logic devices (CPLDs) (e.g., ALTERA MAXII, as an example).A CLD may include task-specific logic such as bus controllers, Ethernetmedia access controllers (MAC), and encryption/decryption modules.External interconnections on a CLD may include serial or parallel datalines or busses. External interconnections may be specialized to supporta particular bus protocol or may be configurable, general-purpose I/Oconnections. Serial and parallel data connections may be implemented viaspecialized hardware or through configured logic blocks.

Memory within a configurable logic device may be arranged in varioustopologies. Many types of configurable logic devices include somearrangement of memory to store configuration information. In somedevices, individual programmable logic units or clusters of such unitsmay include memory blocks. In some devices, one or more larger sharedbanks of memory are provided that are accessible to programmable logicunits via internal interconnections or busses. Some configurable logicdevices may include multiple arrangements of memory.

A configurable logic device may be configured, or programmed, atdifferent times. In some circumstances, a configurable logic device maybe programmed at the time of manufacture (of the configurable logicdevice or of a device containing the configurable logic device). Thismanufacture-time programming may be performed by applying a mask to thedevice and energizing a light or other electromagnetic wave form topermanently or semi-permanently program the device. A configurable logicdevice may also be programmed electronically at manufacture time,initialization time, or dynamically. Electronic programming involvesloading configuration information from a memory or over an input/outputconnection. Some configurable logic devices may include onboardnon-volatile memory (e.g., flash memory) for storing configurationinformation. Such an arrangement allows the configurable logic device toprogram itself automatically when power is applied.

As used herein, the terms processor and CPU mean general purposecomputing devices with fixed instruction sets or microinstruction setssuch as x86 processors (e.g., the INTEL XEON family and the AMD OPTERONfamily, as examples only), POWERPC processors, and other well-knownprocessor families. The terms processor and CPU may also includegraphics processing units (GPUs) (e.g., NVIDIA GEFORCE family, as anexample) and network processors (NPs) (e.g, NETLOGIC XLR and family,INTEL IXP family, CAVIUM OCTEON, for example). Processors and CPUs aregenerally distinguished from CLDs as defined above (e.g., FPGAs, CPLDs,etc.) Some hybrid devices include blocks of configurable logic andgeneral purpose CPU cores (e.g., XILINX VIRTEX family, as an example)and are considered CLDs for the purposes of this disclosure.

An application-specific integrated circuit (ASIC) may be implemented asa processor or CLD as those terms are defined above depending on theparticular implementation.

As used herein, the term instruction executing device means a devicethat executes instructions. The term instruction executing deviceincludes a) processors and CPUs, and b) CLDs that have been programmedto implement an instruction set.

Management switch 110 allows and manages communications among thevarious components of testing architecture 100A, as well ascommunications between components of testing architecture 100A andcomponents of one or more other cards 54 (e.g., via backplane 56 asdiscussed above with respect to FIG. 3). Management switch 110 may be aEthernet layer 2 multi-port switch.

FIG. 5 is a more detailed illustration of the example testing andsimulation architecture 100A shown in FIG. 4, according to an exampleembodiment. As shown, example testing and simulation architecture 100Aincludes controller 106; memory 109 coupled to controller 106; twonetwork processors 105; various CLDs 102 (e.g., capture and offload CLDs102A, router CLDs 102B, and a traffic generation CLD 102C); memorydevices 103A and 103B coupled to CLDs 102A and 102B, respectively;management switch 110 coupled to network processors 105 and CLDs 102A,102B, and 102C, as well as to backplane 56 (e.g., for connection toother cards 54); test interfaces 101 for connecting testing architecture100A to a system 18 to be tested; and/or any other suitable componentsfor providing any of the various functionality of network testing system16 discussed herein or understood by one or ordinary skill in the art.

As discussed above, the components of example architecture 100A may beprovided on a single blade 54, and multiple blades 54 may be connectedtogether via backplane 54 to create larger systems. The variouscomponents of example architecture 100A are now discussed, according toexample embodiments.

Test Interfaces 101

Test interfaces 101 may comprise any suitable communication interfacesfor connecting architecture 100A to a test system 18 (e.g., network 12or device 14). For example, test interfaces 101 may implement Ethernetnetwork connectivity to a test system 18. In one embodiment, interfaces101 may work with SFP+ modules, which allow changing the physicalinterface from 10 Mbps 10-BaseT twisted pair copper wiring to 10 Gbpslong-range fiber. The test interfaces 101 may include one or morephysical-layer devices (PHYa) and SFP+ modules. The PHYs and SFP+modules may be configured using low-speed serial buses implemented bythe capture and offload CLDs 102A (e.g., MDIO and I2C).

Capture and Offload CLDs 102A

An CLD (Field Programmable Gate Array) is a reprogrammable device thatcan be modified to simulate many types of hardware. Beingreprogrammable, it can be continually expanded to offer new accelerationand network analysis functionality with firmware updates. Exampletesting and simulation architecture 100A includes various CLDsdesignated to perform different functions, including two “capture andoffload CLDs” 102A capturing data packets, two “router CLDs” 102B forrouting data between components of architecture 100A, and a trafficgeneration CLD 102C for generating traffic that is delivered to the testsystem 18.

The capture and offload CLDs 102A have the following relationships toother components of testing and simulation architecture 100A:

1. Each capture and offload CLDs 102A is connected to one or more testinterfaces 101. Thus, CLDs 102A are the first and last device in thepacket-processing pipeline. In some embodiments, Ethernet MACs (MediaAccess Controllers) required to support 10/100/1000 and 10000 MbpsEthernet standards are implemented within CLDs 102A and interact withthe physical-layer devices (PHYs) that implement with the testinterfaces 101.

2. Each capture and offload CLDs 102A is also connected to a capturememory device 103A that the CLD 102A can write to and read from. Forexample, each CLD 102A may write to capture memory 103 when capturingnetwork traffic, and read from memory 103 when performing captureanalysis and post-processing.

3. Each capture and offload CLDs 102A is connected to the trafficgeneration CLD 102C. In this capacity, the CLDs 102A is a pass-throughinterface; packets sent by the traffic generation CLD 102C are forwardeddirectly to an Ethernet test interface 101 for delivery to the testsystem 18

4. Each capture and offload CLDs 102A is connected to a router CLD 102Bfor forwarding packets to and from the NPs (105) and the controller 106.

5. Each capture and offload CLDs 102A is connected to the managementswitch 110 which allows for configuration of the CLD 102A and dataextraction (in the case of capture memory 103) from the controller 106or a network processor 105.

Each capture and offload CLDs 102A may be programmed to implement thefollowing functionality for packets received from test interfaces 101.First, each capture and offload CLD 102A may capture and store a copy ofeach packet received from a test interface 101 in the capture memory 103attached to CLD 102A, along with a timestamp for when that packetarrived. Simultaneously, the capture and offload CLD 102A may determineif the packet was generated originally by the traffic generation CLD102C or some other subsystem. If CLD 102A determines that the packet wasgenerated originally by the traffic generation CLD 102C, the CLD 102Acomputes receive statistics for the high-speed packet generation andmeasurement subsystem 22 of system 16 (e.g., refer to FIG. 1). In someembodiments, the packet is not forwarded to any other subsystem in thiscase. Alternatively, if capture and offload CLD 102A determines that apacket was not generated originally by the traffic generation CLD 102C,the capture and offload CLD 102A may parse the packet's layer 2/3/4headers, validate all checksums (up to 2 layers), insert a receivetimestamp, and forward the packet to the closest router CLD 102B forfurther processing.

Each capture and offload CLDs 102A may also be programmed to implementthe following functionality for packets that it transmits to a testinterface 101 for delivery to the test system 18. Packets received at acapture and offload CLD 102A from the traffic generation CLD 102C areforwarded by the CLD 102A as-is to the test interface 101 for deliveryto the test system 18. Packets received at a capture and offload CLD102A from a router CLD 102B may have instructions in the packet forspecific offload operations to be performed on that packet before it issent out trough a test interface 101. For example, packets may includeinstructions for any one or more of the following offload operations:(a) insert a timestamp into the packet, (b) calculate checksums for thepacket on up to 2 layers of IP and TCP/UDP/ICMP headers, and/or (c)split the packet into smaller TCP segments via TCP segmentation offload.Further, a capture and offload CLD 102A may forward a copy of eachpacket (or particular packets) for storage in the capture memory 103Battached to the CLD 102A, along with a timestamp indicating when eachpacket was sent.

In addition to forwarding packets out a test interface 101, each captureand offload CLD 102A may be configured to “simulate” a packet being sentand instead of actually transmitting the packet physically on a testinterface 101. This “loopback” mode may be useful for calibratingtimestamp calculations for the rest of architecture 100A or system 16 byproviding a fixed, known latency on network traffic. It may also beuseful for debugging hardware and network configurations.

Capture Memory 103

As discussed above, each capture and offload CLDs 102A may be connectedto capture memory device 103A that the CLD 102A can write to and readfrom. Capture memory device 103A may comprise any suitable type ofmemory device, e.g., DRAM, SRAM, or Flash memory, hard dive, or anyother memory device with sufficient bandwidth. In some embodiments, ahigh-speed double data rate SDRAM (e.g., DDR2 or DDR3) memory interfaceis provided between each capture and offload CLDs 102A and itscorresponding capture memory device 103A. Thus, data may be written atnear maximum-theoretical rates to maintain an accurate representation ofall packets that arrived on the network, within the limits of the amountof available memory.

Router CLDs 102B

Router CLDs 102B may have similar flexibility as the capture and offloadCLD 102A. Router CLDs 102B may implement glue logic that allows thenetwork processors 105 and controller 106 the ability to send andreceive packets on the test network interfaces 101. Each router CLD 102Bmay have the following relationships to other components of testing andsimulation architecture 100A:

1. Each router CLD 102B is connected to a capture and offload CLD 102A,which gives it a set of “local” test interface (e.g., Ethernetinterfaces) 101 with which it can send and receive packets.

2. The router CLDs 102B are also connected to each other by aninterconnection 120. Thus, packets can be sent and received on “remote”test interfaces 101 via an interconnected router CLD 102B. For example,the router CLDs 102B shown on the right side of FIG. 5 may send andreceive packets via the test interface 101 shown on the left side ofFIG. 5 by way of interconnection 120 between the two CLDs 102B.

3. A network processor 105 may connect to each router CLD 102B via twoparallel interfaces 122 (e.g., two parallel interfaces 10 gigabitinterfaces). These two connections may be interleaved to optimizebandwidth utilization for network traffic. For example, they may be usedboth for inter-processor communication (e.g., communications betweennetwork processors 105 and between controller 106 and network processors105) and for sending traffic to and from the test interfaces 101.

4. Controller 106 also connects to each router CLD 102B. For example,controller 106 may have a single 10 gigabit connection to the eachrouter CLD 102B, which may serve a similar purpose as the networkprocessor connections 122. For example, they may be used both forinter-processor communication and for sending traffic to and from thetest interfaces 101.

5. Each router CLD 102B may include a high-speed, low-latency SRAMmemory. This memory may be used for storing routing tables, statistics,TCP reassembly offload, or other suitable data.

6. Each router CLD 102B is connected to the management switch 110, whichmay allow for configuration of the router CLD 102B and extraction ofstatistics, for example.

In some embodiments, for packets sent from a network processor 105 orcontroller 106, the sending processor 105, 106 first specifies a targetaddress in a special internal header in each packet. This address mayspecify a test interface 101 or another processor 105, 106. The routerCLD 102B may use the target address to determine where to send thepacket next, e.g., it may direct the packet to the another router CLD102B or to the nearest capture and offload CLD 102A.

For incoming packets from the test system 18 that arrive at a router CLD102B, more processing may be required, because the target address headeris absent for packets that have arrived from the test system 18. In someembodiments, the following post-processing is performed by a router CLD102B for each incoming packet from the test system 18:

1. The router CLD 102B parses the packet is parsed to determine the VLANtag and destination IP address of the packet.

2. The router CLD 102B consults a programmable table of IP addresses(e.g., implemented using memory built-in to the CLD 102B) to determinethe address of the target processor 105, 106. This contents of thistable may be managed by software of controller 106.

3. The router CLD 102B computes a hash function on the source anddestination IP addresses and port numbers of the packet.

4. The router CLD 102B inserts a 32-bit hash value into the packet(along with any latency, checksum status, or other offload informationinserted by the respective offload and capture CLD 102A).

5. The router CLD 102B then uses the hash value to determine the optimalphysical connection to use for a particular processor address (because anetwork processor 105 has two physical connections 122, as shown in FIG.5).

6. If the packet is not IP, has no matching VLAN, or has no otherspecific routing information, the router CLD 102B consults a series of“default” processor addresses in an auxiliary table (e.g., implementedusing memory built-in to the CLD 102B).

In some embodiments, the router CLD 102B also implements TCP reassemblyoffloads and extra receive buffering using attached memory (e.g.,attached SRAM memory). Further, it can be repurposed for any othersuitable functions, e.g., for statistics collection by network processor105.

Network Processors 105

Each network processor (NP) 105 may be a general purpose CPU withmultiple cores, security, and network acceleration engines. In someembodiments, each network processor 105 may be an off-the-shelfprocessor designed for network performance. However, it may be veryflexible, and may be suitable to perform tasks ranging from low-level,high-speed packet generation to application and user-level simulation.Each network processor 105 may have the following relationships to othercomponents of testing and simulation architecture 100A:

1. Each network processor 105 may be connected to a router CLD 102B. Therouter CLD 102B may provide the glue logic that allows the processor 105to send and receive network traffic to the rest of the system and outthe test interfaces 101 to the test system 18.

2. Each network processor 105 may be also connected to the managementswitch 110. In embodiments in which the network processor 105 has nolocal storage (e.g. a disk drive), it may load its operating system andapplications from the controller 106 via the management network. As usedherein, the “management network” includes management switch 110, CLDs102A, 102B, and 102C, backplane 56, and controller 106.

3. Because the CLDs 102 are all connected to the management switch 110,the network processors 105 may be responsible for managing andconfiguring certain aspects of the router CLDs 102B and offload andcapture CLDs 102A.

In some embodiments, each network processor 105 may also have thefollowing high-level responsibilities:

1. The primary TCP/IP stack used for network traffic simulation executeson the network processor 105.

2. IP and Ethernet-layer address allocation and routing protocols arehandled by the network processor 105.

3. User and application-layer simulation also run on the networkprocessor 105.

4. The network processor 105 works with software on the controller 106to collect statistics, which may subsequently be used by the statisticsand reporting engine 162 of subsystem 28.

5. The network processor 105 may also collect statistics from CLDs 102A,102B, and 102C and report them to the controller 106. In an alternativeembodiment, the controller 106 itself is configured to collectstatistics directly from CLDs 102A, 102B, and 102C.

Controller 106

Controller 106 may compare any suitable controller programmed to controlvarious functions of system architecture 100A. In some embodiments,controller 106 may be a general purpose CPU with multiple cores, withsome network but no security acceleration. For example, controller 106may be an off-the-shelf processor designed primarily for calculationsand database performance. However, it can also be used for other tasksin the system 100A, and can even be used as an auxiliary networkprocessor due to the manner in which it is connected to the system.Controller 106 may have the following relationships to other componentsof testing and simulation architecture 100A:

1. Controller 106 manages a connection with a removable disk storagedevice 109 (or other suitable memory device).

2. Controller 106 may connect to the management switch 110 to configure,boot, and manage all other processors 105 and CLDs 102 in the system100A.

3. Controller 106 is connected to each router CLD 102B for the purposeof high-speed inter-processor communication with network processors 105(e.g., to provide a 10 Gbps low-latency connection to the networkprocessors 105 in addition to the 1 Gbps connection provided via themanagement switch 110), as well as generating network traffic via testinterfaces 101.

Controller 106 may be the only processor connected directly to theremovable disk storage 109. In some embodiments, all firmware orsoftware used by the rest of the system 100A, except for firmwarerequired to start the controller 106 itself (BIOS) resides on the diskdrive 109. A freshly manufactured system 100A can self-program all othersystem components from the controller 106.

In some embodiments, controller 106 may also have the followinghigh-level responsibilities:

1. Controller 106 serves the user-interface (web-based) used formanaging the system 100A.

2. Controller 106 runs the middle-ware and server applications thatcoordinates the rest of the system operation.

3. Controller 106 serves the operating system and application files usedby network processors 105.

4. Controller 106 hosts the database, statistics and reporting engine162 of statistics collection and reporting subsystem 28.

Traffic Generation CLD 102C

The of the traffic generation CLD 102C is to generate traffic atline-rate. In some embodiment, traffic generation CLD 102C is configuredto generate layer 2/layer 3 traffic; thus, traffic generation CLD 102Cmay be referred to as an L2/L3 traffic CLD.

In an example embodiment, traffic generation CLD 102C is capable ofgenerating packets at 10 Gbps, using a small packet size (e.g., thesmallest possible packet size), for the four test interfaces 101simultaneously, or 59,523,809 packets per second. In some embodiments,this functionality may additionally or alternatively be integrated intoeach capture and offload CLD 102A. Traffic generation CLD 102C may havethe following relationship to other components of testing and simulationarchitecture 100A:

1. Traffic generation CLD 102C is connected to capture and offload CLDs102A. For example, traffic generation CLD 102C may be connected tocapture and offload CLDs 102A via two 20 Gbps bi-directional links.Traffic generation CLD 102C typically only sends traffic, but is mayalso be capable of receiving traffic or other data.

2. Traffic generation CLD 102C is connected to the management switch 110which allows for configuration of CLD 102C for generating traffic.Controller 106 may be programmed to configure traffic generation CLD102C, via management switch 110.

Like other CLDs, traffic generation CLD 102C is reconfigurable and thusmay be reconfigured to provide other functions as desired.

Buffer/Reassembly Memory 103B

A buffer/reassembly memory device 103B may be coupled to each routerCLDs 102B. Each memory device 103B may comprise any suitable memorydevice. For example, each memory device 103B may comprise high-speed,low-latency QDR (quad data rate) SRAM memory attached to thecorresponding router CLD 103B for various offload purposes, e.g.,statistics collection, packet buffering, TCP reassembly offload, etc.

Solid State Disk Drive 109

A suitable memory device 109 may be coupled to controller 106. Forexample, memory device 109 may comprise a removable, solid-state drive(SSD) in a custom carrier that allows hot-swapping and facilitateschanging software or database contents on an installed board. Disk drive109 may store various data, including for example:

1. Firmware that configures the CLDs 102 and various perhipherals;

2. An operating system, applications, and statistics and reportingdatabase utilized by the controller 106; and

3. An operating system and applications used by each network processor105.

Management Switch 110

The management switch 110 connects to every CLD 102, network processor105, and control CPU 106 in the system 100A. In some embodiments,management switch 110 comprises a management Ethernet switch configuredto allow communication of for 1-10 Gbit traffic both between blades 54and between the various processors 105, 106 and CLDs 102 on eachparticular blade 54. Management switch 110 may route packets based onthe MAC address included in each packet passing through switch 110.Thus, management switch 110 may essentially act as a router, allowingcontrol CPUs 106 to communication with network processor 105 and CLD 102on the same card 54 and other cards 54 in the system 16. In suchembodiment, all subsystems are controllable via Ethernet, such thatadditional processors and CLDs may be added by simply chainingmanagement switches 110 together.

In an alternative embodiment, control CPU 106 of different cards 54 maybe connected in any other suitable manner, e.g., by a local bus or PCI,for example. However, in some instances, Ethernet connectivity mayprovide certain advantages over a local bus or PCI, e.g., Ethernet mayfacilitate more types of communication between more types of devicesthan a local bus or PCI.

Backplane 56

Network testing system 16 may be configured to support any suitablenumber of cards or blades 54. In one embodiment, system 16 is configuredto support between 1 and 14 cards 54 in a single chassis 50. Backplane56 may provide a system for interconnecting the management Ethernetprovided by the management switches 110 of multiple cards 54, as well assystem monitoring connections for measuring voltages and temperatures oncards 54, and for debugging and monitoring CPU status on all cards 54,for example. Backplane 56 may also distribute clock signals between allcards 54 in a chassis 50 so that the time stamps for all CPUs and CLDsremain synchronized.

Network Testing Subsystems and System Operation

In some embodiments, network testing system 16 may provide an integratedsolution that provides some or all of the following functions: (1)high-speed, high-resolution network packet capture, (2) high-speedpacket generation and measurement, (3) application-level simulation andmeasurement, (4) security and exploit simulation and analysis, and (5)statistics collection and reporting. Thus, as discussed above withrespect to FIG. 1, network testing system 16 may comprise a high-speed,high-resolution network packet capture subsystem 20, a high-speed packetgeneration and measurement subsystem 22, an application-level simulationand measurement subsystem 24, a security and exploit simulation andanalysis subsystem 26, and/or a statistics collection and reportingsubsystem 28. The architecture of system 16 (e.g., example architecture100A discussed above or example architecture 100B discussed below) mayallow for some or all of these subsystems 20-28 to operatesimultaneously and cooperatively within the same software and hardwareplatform. Thus, system 16 may be capable of generating and analyzingpackets at line rate, while simultaneously capturing that same traffic,performing application simulation and security testing.

FIGS. 6A-10 illustrates the relevant components and method flowsprovided by each respective subsystem 20-28. In particular, FIGS. 6A and6B illustrate relevant components and an example process flow providedby high-speed, high-resolution network packet capture subsystem 20;FIGS. 7A and 7B illustrate relevant components and an example processflow provided by high-speed packet generation and measurement subsystem22; FIGS. 8A and 8B illustrate relevant components and an exampleprocess flow provided by application-level simulation and measurementsubsystem 24; FIGS. 9A and 9B illustrate relevant components and anexample process flow provided by security and exploit simulation andanalysis subsystem 26; and FIG. 10 illustrate relevant components ofstatistics collection and reporting subsystem 28. The components of eachsubsystem 20-28 correspond to the components of example architecture100A shown in FIGS. 4 and 5. However, it should be understood that eachsubsystem 20-28 may be similarly implemented by any other suitablesystem architecture, e.g., example architecture 100B discussed belowwith reference to FIG. 15.

High-Speed, High-Resolution Network Packet Capture Subsystem 20

Modern digital networks involve two or more or nodes that send databetween each other over a shared, physical connection using units ofdata called packets. Packets contain information about the source anddestination address of the nodes, application information. A networkpacket capture is the observing and storage of packets on the networkfor later debugging and analysis.

Network packet capture may be performed for various reasons, e.g.,lawful intercept (tapping), performance analysis, and applicationdebugging, for example. Packet capture devices can range in complexityfrom a simple desktop PC (most PCs have limited capture abilities builtinto their networking hardware) to expensive purpose-built hardware.These devices vary in both their capacity and accuracy. A limitedcapture system is typically unable to capture all types of networkpackets, or sustain capture at the maximum speed of the network.

In contrast, network packet capture subsystem 20 of network testingsystem 16 may provide high-speed, high-resolution network packet capturecapable of capturing all types of network packets (e.g., Ethernet, TCP,UDP, ICMP, IGMP, etc.) at the maximum speed of the tested system 18(e.g., 4.88 million packets per second, transmit and receive, per testinterface).

FIG. 6A illustrates relevant components of subsystem 20. In an exampleembodiment, network packet capture subsystem 20 may utilize thefollowing system components:

(a) One or more physical Ethernet test interface (PHY) 101.

(b) An Ethernet MAC (Media Access Controller) 130 implemented inside CLD102A per physical interface 101 which can be programmed to enter“promiscuous mode,” in which the Ethernet MAC can be instructed to snoopall network packets, even those not addressed for it. Normally, anEthernet MAC will only see packets on a network that include its localMAC Address, or that are addressed for “broadcast” or “multicast”groups. A MAC Address may be a 6-byte Ethernet media access controladdress. A capture system should be able to see all packets on thenetwork, even those that are not broadcast, multicast, or addressed withthe MAC's local MAC address. In some embodiments, it may be desirable toenter a super-promiscuous mode in order to receive even “erroneous”packets. Typical Ethernet MACs will drop malformed or erroneous packetseven if in promiscuous mode on the assumption that a malformed orerroneous packet is likely damaged and the sender should resend acorrect packet if the message is important. These packets may be ofinterest in a network testing device such as system 16 to identify anddiagnose problem connections, equipment, or software. Thus, the EthernetMAC of CLD 102A may be configured to enter super-promiscuous mode inorder to see and capture all packets on the network, even including“erroneous” packets (e.g., corrupted packets as defined by Ethernet FCSat end of a packet).

(c) A capture and offload CLD 102A.

(d) Capture memory 103A connected to CLD 102A.

(e) Controller software 132 of controller 106 configured to start, stopand post-process packet captures.

(f) A management processor 134 of controller 106 configured to executethe controller software 132.

(g) Management switch 110 configured to interface and control thecapture and offload CLD 102A from the management processor 134.

An example network packet capture process is now described. When thepacket capture feature is enabled by a user via the user interfaceprovided by the system 100A (see FIG. 13C), controller 106 may configurethe Ethernet MACs 130 and PHYs 101 to accept all packets on the network,i.e., to enter “promiscuous mode.” Controller 106 may then configure thecapture and offload CLD 102A to begin storing all packets sent orreceived via the Ethernet MAC/PHY in the high-speed capture memory 103Aattached to the CLD 102A. When the Ethernet MAC/PHY sends or receives apacket, it is thus captured in memory 103A by CLD 102A. For eachcaptured packet, CLD 102A also generates and records a high-resolution(e.g., 10 nanosecond) timestamp in memory 103A with the respectivepacket. This timestamp data can be used to determine network attributessuch as packet latency and network bandwidth utilization, for example.

Using the architecture discussed herein, system 16 can store packetssent and received at a rate equivalent to the maximum rate possible onthe network. Thus, as long as there is sufficient memory 103A attachedto the CLD 102A, a 100% accurate record of the traffic that occurred ontest system 18 may be recorded. If memory 103A fills up, a wrappingmechanism of CLD 102A allows CLD 102A to begin overwriting the oldestpackets in memory with newer packets.

To achieve optimal efficiency, CLD 102A may store packets in memory intheir actual length and may use a linked-list data structure todetermine where the next packet begins. Alternatively, CLD 102A mayassume all packets are a fixed size. While this alternative iscomputationally efficient (a given packet can be found in memory bysimply multiplying by a fixed value), memory space may be wasted whenpackets captured on the network are smaller than the assumed size.

CLD 102A may also provide a tail pointer that can be used to walkbackward in the list of packets to find the first captured packet. Oncethe first captured packet is located, the control software 132 can readthe capture memory 103A and generate a diagnostic file, called a PCAP(Packet CAPture) file, which can be sent to the user and/or stored indisk 109. This file may be downloaded and analyzed by a user using athird-party tool.

Because there can be millions of packets in the capture memory 103A,walking through all of the packets in the packet capture to located thefirst captured packet based on the tail pointer may take considerabletime. Thus, CLD 102A may provide a hardware-implementation that walksthe linked list and can provide the head pointer directly. In addition,copying the capture memory 103A to a file that is usable for analysiscan take additional time. Thus, CLD 102A may implement abulk-memory-copy mode that speeds up this process.

FIG. 6B illustrates an example network packet capture process flow 200provided by subsystem 20 shown in FIG. 6A and discussed above. At step202, controller 106 may configure the capture and offload CLD 102A andtest interfaces 101 to begin packet capture, e.g., as discussed above.At step 204, the packet capture may finish. Thus, at step 206,controller 106 may configure CLD 102A and test interfaces 101 to stoppacket capture.

At step 208, CLD 102A may rewind capture memory 103A, e.g., using tailpointers as discussed above, or using any other suitable technique. Atstep 210, controller 106 may read data from capture memory 103A andwrite to disk 109, e.g., in the form of a PCAP (Packet CAPture) file asdiscussed above, which file may then be downloaded and analyzed usingthird-party tools.

Table 1 provides a comparison of the performance of network packetcapture subsystem 20 to certain conventional solutions, according to anexample embodiment of system 16.

TABLE 1 Network packet Conventional Conventional capture subsystem 20desktop PC dedicated solution Storage RAM (4 GB) disk disk medium(high-speed) Timestamp nanoseconds milliseconds nanoseconds resolutionSpeed per 14M pps 100k pps Millions of pps interface

In some embodiments, dedicated packet capture memory and hardware may beomitted, e.g., for design simplicity, cost, etc. In such embodiments, asoftware-only implementation of packet capture may instead be provided,although such implementation may have reduced performance as comparedwith the dedicated packet capture memory and hardware subsystemdiscussed above.

High-Speed Packet Generation and Measurement Subsystem 22

Modern networks can transport packets at a tremendous rate. A comparisonof various network speeds and the maximum packets/second that they canprovide is set forth in Table 2.

TABLE 2 Network speed Era Maximum packets/second 10 Mbps Ethernet early1990s 14,880 packets/sec 100 Mbps Ethernet late 1990s 148,809packets/sec 1 Gbps Ethernet early 2000s 1,488,095 packets/sec 10 GbpsEthernet late 2000s 14,880,952 packets/sec 40 Gbps Ethernet early 2010s59,523,809 packets/sec 100 Gbps Ethernet early 2010s 148,809,523packets/sec

The data rate for the fastest network of a given era typically exceedsthe number of packets/second that a single node on the network canpractically generate. Thus, to test the network at its maximum-possiblepacket rate, one might need to employ either many separate machines, ora custom solution dedicated to generating and receiving packets at thehighest possible rate.

As discussed above, network testing system 16 may include a high-speedpacket generation and measurement subsystem 22 for providing packetgeneration and measurement at line rate. FIG. 7A illustrates relevantcomponents of subsystem 22. In an example embodiment, high-speed packetgeneration and measurement subsystem 22 may utilize the following systemcomponents:

(a) One or more physical Ethernet test interface (PHY) 101.

(b) An Ethernet MAC (Media Access Controller) 130 on capture and offloadCLD 102A per physical interface 101.

(c) An L2/L3 traffic generation CLD 102C configured to generate packetsto be sent to the Ethernet MAC 130.

(d) A capture and offload CLD 102A configured to analyze packets comingfrom the Ethernet MAC 130.

(e) Controller software 132 of controller 106 configured to generatedifferent types of network traffic.

(f) Controller software 132 of controller 106 configured to managenetwork resources, allowing the CLD-generated traffic to co-exist withthe traffic generated by other subsystems, at the same time.

(g) A management processor 134 of controller 106 configured to executethe controller software 132.

The CLD solution provided by subsystem 22 is capable of sending trafficand analyzing traffic at the maximum packet rate for 10 Gbps Ethernet,which may be difficult for even a high-end PC. Additionally, subsystem22 can provide diagnostic information at the end of each packet itsends. This diagnostic information may include, for example:

1. a checksum (e.g., CRC32, for verifying packet integrity);

2. a sequence number (for determining if packets were reordered on thenetwork);

3. a timestamp (for determining how long the packet took to traverse thenetwork); and/or

4. a signature (for uniquely distinguishing generated traffic from othertypes of traffic).

The checksum may be placed at the end of each packet. This checksumcovers a variable amount of the packet, because as a packet traversesthe network, it may be expected to change in various places (e.g., thetime-to-live field, or the IP addresses). The checksum allowsverification that a packet has not changed in unexpected ways or beencorrupted in-transit. In some embodiments, the checksum is a 32-bit CRCchecksum, which is more reliably able to detect certain types ofcorruption that the standard 16-bit 2's complement TCP/IP checksums.

The sequence number may allow detection of packet ordering even if thenetwork packets do not normally have a method of detecting the sequencenumber. This sequence number may be 32-bit, which wraps less quickly ona high-speed network as compared to other standardized packetidentifiers, e.g., the 16-bit IP ID.

The timestamp may have any suitable resolution. For example, thetimestamp may have a 10 nanosecond resolution, which is fine-grainedenough to measure the difference in latency between a packet travelingthrough a 1 meter and a 20 meter optical cable (effectively measuringthe speed of light.)

The signature field may allows the CLD 102A to accurately identifypackets that need analysis from other network traffic, without relyingon the simulated packets having any other identifiable characteristics.This signature also allows subsystem 22 to operate without interferingwith other subsystems while sharing the same test interfaces 101.

FIG. 7B illustrates an example network packet capture process flow 220provided by subsystem 22 shown in FIG. 7A and discussed above. At step222, controller 106 may configure traffic generation CLD 102C, captureand offload CLDs 102A, and test interfaces 101 to begin packetgeneration and measurement. At step 224, controller 106 may collectstatistics from capture and offload CLDs 102A related to the kind andquantity of network traffic that was generated and received, and storethe statistics in disk 109. At step 226, the test finishes. Thus, atstep 228, controller 106 may configure traffic generation CLD 102C,capture and offload CLDs 102A, and test interfaces 101 to stop packetgeneration and measurement. At step 230, a reporting engine 162 oncontroller 106 may generate reports based on data collected and storedat step 224.

Application-Level Simulation and Measurement Subsystem 24

While high-speed packet generation and analysis can be used toillustrate raw network capacity, integrity and latency, modern networksalso analyze traffic beyond individual packets and instead look atapplication flows. This is known as deep packet inspection. Also, it isoften desired to measure performance of not only the network itself butindividual devices, such as routers, firewalls, load balancers, servers,and intrusion detection and prevention systems, for example.

To properly exercise these systems, higher-level application data issent on top of the network. Network testing system 16 may include anapplication-level simulation and measurement subsystem 24 to providesuch functionality. FIG. 8A illustrates relevant components of subsystem24. In an example embodiment, application-level simulation andmeasurement subsystem 24 may utilize the following system components:

(a) One or more physical Ethernet test interface (PHY) 101.

(b) An Ethernet MAC (Media Access Controller) 130 capture and offloadCLD 102A per physical interface 101.

(c) Multiple network processors 105 configured to generate and analyzehigh-level application traffic.

(d) Multiple capture and offload CLDs 102A and router CLDs 102Bconfigured to route traffic between the Ethernet MACs 130 and thenetwork processors 105 and to perform packet acceleration offload tasks.

(e) Software 142 of network processor 105 configured to generateapplication traffic and generate statistics.

(f) Controller software 132 of controller 106 to manage networkresources, allowing the network processor-generated application trafficto co-exist with the traffic generated by other subsystems, at the sametime.

(g) A management processor 134 on controller 106 configured to executethe controller software 132.

Application-Level Simulation: Upper Layer

In some embodiments, the network processors 105 execute software 142that implements both the networking stack (Ethernet, TCP/IP, routingprotocols, etc.) and the application stack that is typically present ona network device. In this sense, the software 142 can simulate networkclients (e.g., Desktop PCs), servers, routers, and a whole host ofdifferent applications. This programmable “application engine” software142 is given instructions on how to properly simulate a particularnetwork or application by an additional software layer. This softwarelayer may provide information such as:

1. Addresses and types of hosts to simulate on the network,

2. Addresses and types of hosts to target on the network,

3. Types of applications to simulate, and/or

4. Details on how to simulate a particular application (mid-levelinstructions for application interaction).

The details on how to simulate applications reside in software 144 thatruns on the management processor 134 on controller 106. A user can modelan application behavior in a user interface (see, e.g., FIGS. 13A-13D)that provides high-level application primitives, such as to make adatabase query or load a web page, for example. These high-levelbehaviors are translated by software 144 into low-level instructions,such as “send a packet, expect 100 bytes back,” which are then executedby the application engine 142 running on the network processor 105. Newapplications can be implemented by a user (e.g., a customer or in-housepersonnel), without any changes to the application engine 142 itself.Thus, it is possible to add new functionality without upgradingsoftware.

Application-Level Simulation: Lower Layer

Physically, the network processors 105 connect to multiple CLDs 102. Allpackets that leave the network processor 105 first pass through one ormore CLDs 102 before they are sent to the Ethernet interfaces 101, andall packets that arrive via the Ethernet interfaces 101 pass through oneor more CLDs 102 before they are forwarded to a network processor 105.The CLDs 102 are thus post- and pre-processors for all network processortraffic. In addition, the packet capture functionality provided bysubsystem 20 (discussed above) is able to capture all networkprocessor-generated traffic.

The CLDs 102A and 102B may be configured to provide some or all of thefollowing additional functions:

1. Programmable timestamp insertion and measurement (by CLD 102A),

2. TCP/IP Checksum offload (by CLD 102A),

3. TCP segmentation offload (by CLD 102A), and/or

4. Incoming packet routing and load-balancing, to support multiplenetwork processors using the same physical interface (by CLD 102B).

For timestamp insertion, a network processor 105 can request that theCLD 102A insert a timestamp into a packet originally generated by thenetwork processor 105 before it enters the Ethernet. The CLD 102A canalso supply a timestamp for when a packet arrives before it is forwardedto a network processor 105. This is useful for measuringhigh-resolution, accurate packet latency in a way typically onlyavailable to a simple packet generator on packets containing realisticapplication traffic. Unlike conventional off-the-shelf hardware that caninsert and capture timestamps, CLD 102A is configured to insert atimestamp into any type of packet, including any kind of packets, e.g.,PTP, IP, TCP, UDP, ICMP, or Ethernet-layer packets, instead of only PTP(Precision Time Protocol) packets as part fo the IEEE 1588 standard.

TCP/IP checksum offload may also be performed by the CLDs 102A. Unlike atypical hardware offload implemented by an off-the-shelf Ethernetcontroller, the CLD implementation of system 16 has an additionalfeature in that any packet can have multiple TCP/IP checksums computedby CLD 102A on more than one header layer in the packet. This may beespecially useful when generating packets that are tunneled, and thushave multiple TCP, IP or UDP checksums. Conventional solutions cannotperform a checksum on more than one header layer in a packet.

For TCP segmentation offload, a single large TCP packet canautomatically be broken into smaller packets by CLD 102A to fit themaximum transmission unit (MTU) of the network. TCP segmentation offloadcan save a great deal of CPU time when sending data at high speeds.Conventional solutions are typically implemented without restrictions,such as all offloaded TCP segments will have the same timestamp. Incontrast, the CLD implementation of system 16 allows timestamping ofindividual offloaded TCP segments as if they had been sent individuallyby the network processor 105.

Incoming packet routing and load balancing enable multiple networkprocessors 105 to be used efficiently in a single system. Conventionalload-balancing systems rely on some characteristic of each incomingpacket to be unique, such as the IP or Ethernet address. In the eventthat the configured attributes for incoming packets are not unique, asystem can make inefficient use of multiple processors, e.g., alltraffic goes to one processor rather than being fairly distributed. Incontrast, the CLD 102B implementation of packet routing in system 16provides certain features not typically available in commodity packetdistribution systems such as TCAMs or layer-3 Ethernet switches. Forexample, the CLD 102B implementation of system 16 may provide any one ormore of the following features:

1. The CLD implementation of system 16 can be reconfigured to parsepackets two headers deep. If all traffic has a single outer header,e.g., tunneled traffic, the system can look further to find uniqueidentifiers in the packets.

2. The system 16 may employ a hardware implementation of jhash (ahashing algorithm designed by Bob Jenkens, available athttp://burtleburtle.net/bob/c/lookup3.c) to distribute packets, which isharder to defeat than other common implementation such as CRC andefficiently distributes packets that differ by very few bits.

3. Packets can be routed on thousands of arbitrary IP ranges as wellusing a lookup table built into the CLDs 102B.

FIG. 8B illustrates an example network packet capture process flow 240provided by application-level simulation and measurement subsystem 24shown in FIG. 8A and discussed above. At step 242, controller 106 mayconfigure network processors 105, traffic generation CLD 102C, captureand offload CLDs 102A, and test interfaces 101 for a desiredapplication/network simulation. A network processor 105 may then begingenerating network traffic, which is delivered via test interfaces 101to the test system 18. At step 244, the network processor 105 may sendstatistics from itself and from CLDs 102A and 102B to controller 106 forstorage in disk drive 109. Controller 106 may dynamically modifysimulation parameters of the network processor 105 during thesimulation.

At step 246, the simulation finishes. Thus, at step 248, the networkprocessor 105 stops simulation, and controller 106 stops data collectionregarding the simulation. At step 250, the reporting engine 162 oncontroller 106 may generate reports based on data collected and storedat step 244.

Security and Exploit Simulation and Analysis Subsystem 26

In both isolated networks and the public Internet, vulnerable users,applications and networks continue to be exploited in the form ofmalware (virus, worms), denial of service (DoS), distributed denial ofservice (DDoS), social engineering, and other forms of attack. Networktesting system 16 may be configured to generate and deliver malicioustraffic to a test system 18 at the same time that it generates anddelivers normal “background” traffic to test system 18. In particular,security and exploit simulation and analysis subsystem 26 of system 16may be configured to generate such malicious traffic. This may be usefulfor testing test system 18 according to various scenarios, such as forexample:

1. “Needle in a haystack” or lawful intercept testing (i.e., locatingbad traffic among good traffic),

2. Testing the effectiveness of intrusion prevention/detectionmechanisms, and/or

3. Testing the effectiveness of intrusion prevention/detectionmechanisms under load.

FIG. 9A illustrates relevant components of security and exploitsimulation and analysis subsystem 26, according to an exampleembodiment. In this embodiment, subsystem 26 may utilize the followingsystem components:

(a) One or more physical Ethernet test interface (PHY) 101.

(b) An Ethernet MAC (Media Access Controller) 130 implemented in CLD102A (see FIG. 6A) per physical interface 101.

(c) Multiple network processors 105 configured to generate and analyzehigh-level application traffic.

(d) Multiple capture and offload CLDs 102A and router CLDs 102Bconfigured to route traffic between the Ethernet MACs 130 and thenetwork processors 105 and to perform packet acceleration offload tasks.

(e) A “security engine” 150 comprising software 150 configured togenerate malicious application traffic and to verify its effectiveness.Security engine 150 may be provided on a network processor 105 and/orcontroller 106, and is thus indicated by dashed lines in FIG. 9A.

(f) Controller software 132 of controller 106 to manage networkresources, allowing the malicious application traffic to co-exist withthe traffic generated by other subsystems, at the same time.

(g) A management processor 134 on controller 106 configured to executethe controller software 132, collect and store statistics, and/orgenerate malicious application traffic.

As mentioned above, security engine 150 may be provided on a networkprocessor 105 and/or controller 106. For example, in some scenarios, theapplication engine 142 employed by the network processor 105 is used togenerate malicious traffic when high-performance is required. In otherscenarios, the management processor 134 of controller 106 can generatemalicious traffic packet-by-packet and forward these to the networkprocessor 105 as if they were generated locally. This mechanism may beemployed for more sophisticated attacks that do not require highperformance.

FIG. 9B illustrates an example network packet capture process flow 260provided by security and exploit simulation and analysis subsystem 26shown in FIG. 9A and discussed above. At step 262, controller 106 mayconfigure the security engine 150 (running on network processor(s) 105and/or controller 106), network processors 105, traffic generation CLD102C, capture and offload CLDs 102A, and test interfaces 101 withinstructions for a desired security simulation. Security engine 150 maythen begin generating network traffic, which is delivered via testinterfaces 101 to the test system 18. At step 264, security engine 150may send statistics to controller 106 for storage in disk drive 109.Controller 106 may dynamically modify simulation parameters of thesecurity engine 150 during the simulation.

At step 266, the simulation finishes. Thus, at step 268, security engine150 stops simulation, and controller 106 stops data collection regardingthe simulation. At step 270, the reporting engine 162 on controller 106may generate reports based on data collected and stored at step 264.

Statistics Collection and Reporting Subsystem 28

The management processor 134 of controller 106, in addition to providinga place for much of the control software for various subsystems toexecute, may also host a statistics database 160 and reporting engine162. Statistics database 162 both stores raw data generated by othersubsystems as well as derives its own data. For instance, subsystem 20or 22 may report the number and size of packets generated on a networkover time. Statistics database 160 can then compute the minimum,maximum, average, standard deviation, and/or other statistical dataregarding the data rate from these two pieces of data. Reporting engine162 may comprise additional software configured to convert statisticsinto reports including both data analysis and display of the data in anuser-readable format.

FIG. 10 illustrates relevant components of statistics collection andreporting subsystem 28, according to an example embodiment. In thisembodiment, the sub-components of the statistics and reporting subsystem28 may include:

1. A statistics database 160.

2. A storage device 109 to store data collected by other sub-components(e.g. a solid-state flash drive).

3. A data collection engine 164 configured to converts raw data fromsub-components into a normalized form for the database 160.

4. A reporting engine 162 configured to allow analyzing and viewing databoth in real-time and offline.

5. A management processor 134 configured to run the database 160 andsoftware engines 162 and 164.

Reporting engine 162 and data collection engine 164 may comprisesoftware-based modules stored in memory associated with controller 106(e.g., stored in disk 109) and executed by management processor 134.

FIG. 11 illustrates one view of the application system architecture ofsystem 16, according to certain embodiments of the present disclosure.The system architecture may be subdivided into software control andmanagement layer and hardware layers. Functionality may be implementedin one layer or may be implemented across layers.

In the control and management layer, example applications are shownincluding network resiliency, data center resiliency, lawful intercept,scenario editor, and 4G/LTE. Network and data center resiliencyapplications may provide an automated, standardized, and deterministicmethod for evaluating and ensuring the resiliency of networks, networkequipment, and data centers. System 16 provides a standard measurementapproach using a battery of real-world application traffic, real-timesecurity attacks, extreme user load, and application fuzzing. Thatbattery may include a blended mix of application traffic and maliciousattacks, including obfuscations.

Lawful intercept applications may test the capabilities of lawenforcement systems to process realistic network traffic scenarios.These applications may simulate the real-world application traffic thatlawful intercept systems must process—including major Web mail, P2P,VoIP, and other communication protocols—as well as triggering content inmultiple languages. These applications may create needle-in-a-haystackscenarios by embedding keywords to ensure that a lawful interceptsolution under test detects the appropriate triggers; tax theperformance of tested equipment with a blend of application, attack, andmalformed traffic at line rate; and emulate an environment's uniquebackground traffic by selecting from more than tens of applicationprotocols, e.g., SKYPE, VoIP, email, and various instant messagingprotocols.

The scenario editor application may allow modification of existingtesting scenarios or the creation of new scenarios using a rules-basedinterface. The scenario editor application may also enable configurationof scenarios based on custom program logic installed on system 16.

The 4G/LTE application may allow testing and validation of mobilenetworking equipment and systems including mobile-specific services likemobile-specific web connections, mobile device application stores, andother connections over modern wireless channels. These applications maycreate city-scale mobile data simulations to test the resiliency ofmobile networks under realistic application and security traffic. Testsmay measure mobility infrastructure performance and security underextreme network traffic conditions; stress test key LTE networkcomponents with emulation of millions of user devices and thousands oftransmission nodes; and validate per-device accounting, billing, andpolicy mechanisms.

Tcl scripting modules may allow web-based user interface design andconfiguration of existing and user-created applications. Reportingmodules may allow generation of standardized reports on test results,traffic analysis, and ongoing monitoring data.

Supporting those applications is the unified control and test automationsubsystem including two software modules, Tcl scripting and reporting,and three hardware modules, security attacks, protocol fuzzing, andapplication protocols. The latter three modules comprise the applicationand threat intelligence program. Underlying the applications are threehardware layers including security accelerators, network processors, andconfigurable logic devices (CLDs).

Security accelerator modules may provide customizable hardwareacceleration of security protocols and functions. Security attackmodules may provide customizable hardware implementation of specificsecurity attacks that may be timing specific or may require extremelyhigh traffic generation (e.g., simulation of bot-net and denial ofservice attacks). Protocol fuzzying modules may test edge cases innetworking system implementations. A protocol fuzzying module may targeta specific data value or packet type and may generate a variety ofdifferent values (valid or invalid) in turn. The goal of a fuzzer may beto provide malicious or erroneous data or to provide too much data totest whether a device will break and therefore indicate a vunerability.A protocol fuzzying module may also identify constraints bysystematically varying as aspect of the input data (e.g., packet size)to determine acceptable ranges of input data. Application protocolsmodules may provide customizable hardware implementation or testing ofspecific network application protocols to increase overall throughput.

FIG. 12 illustrates one view of select functional capabilitiesimplemented by system 16, according to certain embodiments of thepresent disclosure. Incoming packets, also called ingress packets,arriving on external interfaces may be processed by one or more ofseveral core functional modules in high-speed configurable logicdevices, including:

-   -   Verify IP/TCP Checksums: Checksums provide some indication of        network data integrity and are calculated at various networking        layers including Layer 2 (Ethernet), Layer 3 (Internet        Protocol), and Layer 4 (Transport Control Protocol). Bad        checksums are identified and may be recorded.    -   Timestamp: Timestamps may be used to measure traffic statistics,        correlate captured data with real-time events, and/or to trigger        events such as TCP retransmissions. Ingress packets are each        marked with a high-resolution timestamp upon receipt.    -   Statistics: Statistics may be gathered to monitor various        aspects of systems under test or observation. For example,        response time may be measured as a simulated load is increased        to measure scalability of a device under test.    -   L2/L3 Packet processing: In the process of verifying checksums,        the configurable logic devices may record information (e.g., IP        and TCP packet offsets within the current ingress packet) about        the packet layout to speed later processing.    -   Packet capture/filtering: Many applications benefit from packet        capture into capture memory that allows subsequent analysis of        observed traffic patterns. Filtering may be used to focus the        capture process on packets of particular interest.

The output of one or more of these functional modules, along with VLANprocessing, may be fed into one or more network processors along withthe ingress packet. Likewise, egress packets generated by the networkprocessors may be processed by one or more of several core functionalmodules in high-speed configurable logic devices, including:

-   -   Packet capture/filtering: Many applications benefit from packet        capture into capture memory that allows subsequent analysis of        generated traffic patterns. Filtering may be used to focus the        capture process on packets of particular interest.    -   Statistics: Statistics may be gathered to monitor the output of        system 16. For example, these statistics may be gathered to        analyze the performance of application logic executing on a        network processor or control processor.    -   Generate IP/TCP checksums: Checksum calculation is an expensive        process that may be effectively offloaded to a configurable        logic device for a significant performance gain.    -   Timestamp: A high-resolution timestamp may be added just prior        to transmission to enable precise measurement of response times        of tested systems.    -   TCP segmentation: This process is data and processing intensive        and may be effectively offloaded to a configurable logic device        for a significant performance gain.    -   L2/L3 packet generation: Some types of synthetic network traffic        may be generated by a configurable logic device in order to        maximize output throughput and saturate the available network        channels.

FIG. 13A illustrates user application level interfaces to system 16,according to certain embodiments of the present disclosure. In someembodiments, a workstation (e.g., running a standard operating systemsuch as MAC OSX, LINUX, or WINDOWS) may provide a server for usercontrol and configuration of system 16. In some embodiments, thatworkstation generates a web interface (e.g., via TCL scripts) that maybe accessible via a standard web browser. This web interface maycommunicate with system 16 via an extensible markup language (XML)interface over a secure sockets layer (SSL) connection. In someembodiments, a reporting system may be provided with control process(e.g., one written in the JAVA programming language) mining data from adatabase to generate reports in common formats such as portable documentformat (PDF), WORD format, POWERPOINT format, or EXCEL format.

FIG. 13B illustrates user application level interfaces to system 16,according to certain embodiments of the present disclosure. A controlprocess (e.g., one written in JAVA), may manipulate configuration datain database to control various parameters of system 16. For example,security parameters may configure a RUBY/XML interface to provideindividual access to certain configuration and reporting options. Inanother example, application helper modules may be added and/orconfigured to control application streams on the network processors. Ina further example, network processor configuration parameters may be setto route all application traffic through the network processors. In afinal example, the capture CLD and L2/L3 CLD may be configured tooffload a portion of traffic, e.g., 25%, from the network processors.

FIG. 13C illustrates a user interface screen for configuring aspects ofsystem 16, according to certain embodiments of the present disclosure.Specifically, the screen in FIG. 13C may allow a user to configure theprocess by which captured packet data may be exported at an interval topersistent storage, e.g., on drive 109.

FIG. 13D illustrates a user interface screen for configuring a networktesting application, according to certain embodiments of the presentdisclosure. Specifically, the screen in FIG. 13D may allow a user toconfigure various types of synthetic data flows to be generated bysystem 16. The screen shows the flow type “HTTP Authenticated” asselected and shows the configurable subflows and actions relevant tothat overall flow type.

Specific Example Implementation of Architecture 100A

FIGS. 14A-14B illustrate a specific implementation of the testing andsimulation architecture 100A shown in FIGS. 4 and 5, according to anexample embodiment.

Controller 106 provides operational control of one or more blades inarchitecture 100A. Controller 106 includes control processor 134 coupledto an electrically erasable programmable read only memory (EEPROM)containing the basic input and output system (BIOS), universal serialbus (USB) interfaces 336, clock source 338, joint test action group(JTAG) controller 324, processor debug port 334, random access memory(RAM) 332, and Ethernet medium access controllers (MACs) 330A and 330Bcoupled to non-volatile memories 320/322. EEPROM memory 322 may be usedto store general configuration options, e.g., the MAC address(es), linktypes, and other part-specific configuration options. Flash memory 320may be used to store configurable applications such as network boot(e.g., PXE Boot).

Controller 106 may be an integrated system on a chip or a collection oftwo or more discrete modules. Control processor 134 may be a generalpurpose central processing unit such as an INTEL x86 compatibleprocessor. In some embodiments, control processor 134 may be an INTELXEON processor code-named JASPER FOREST and may incorporate or interfacewith additional chipset components including memory controllers andinput/output controllers, e.g., the INTEL IBEX PEAK south bridge.Control processor is coupled, e.g., via a serial peripheral interface tonon-volatile memory containing BIOS software. (Note that references inthis specification to SPI interfaces, for example those interconnectingCLDs and/or network processors, are references to the system packetinterface (SPI-4.2) rather than the serial peripheral interface.) TheBIOS software provides processor instructions sufficient to configurecontrol processor 134 and any chipset components necessary to accessstorage device 109. The BIOS also includes instructions for loading, orbooting, an operating system from storage device 109 or a USB memorydevice connected to interface 336.

USB interfaces 336 provide external I/O access to controller 106. USBinterfaces 336 may be used by an operator to connect peripheral devicessuch as a keyboard and pointing device. USB interfaces 336 may be usedby an operator to load software onto controller 106 or perform any othernecessary data transfer. USB interfaces 336 may also be used bycontroller 106 to access USB connected devices within system 100A.

Clock source CK505 is a clock source to drive the operation of thecomponents of controller 106. Clock source may be driven by a crystal togenerate a precise oscillation wave.

JTAG controller 324 is a microcontroller programmed to operate as acontroller for JTAG communications with other devices. JTAG provides afallback debugging and programming interface for various systemcomponents. This protocol enables fault isolation and recovery,especially where a device has been incompletely or improperlyprogrammed, e.g., due to loss of power during programming. In certainembodiments, JTAG controller 324 is a CYPRESS SEMICONDUCTOR CY68013microcontroller programmed to execute JTAG instructions and drive JTAGsignal lines. JTAG controller 324 may include or be connected to anon-volatile memory to program the controller on power up.

Processor debug port 334 is a port for debugging control processor 106as well as chipset components. Processor debug port 334 may conform tothe INTEL XDB specification.

RAM 332 is a tangible, computer readable medium coupled to controlprocessor 134 for storing the instructions and data of the operatingsystem and application processes running on control processor 134. RAM332 may be double data rate (DDR3) memory.

Ethernet MACs 330A and 330B provide logic and signal control forcommunicating with standard Ethernet devices. These MACS may be coupledto control processor 134 via a PCIe bus. MACS 330A and 330B may be INTEL82599 dual 10 Gbps parts. In some embodiments, MACs 330A and 330B may beincorporated into control processor 134 or the chipset devices. EthernetMACs 330A and 330B are coupled to non-volatile memories 320/322.

Controller 106 is coupled to tangible, computer readable medium in theform of mass storage device 109, e.g., a solid state drive (SSD) basedon high speed flash memory. In some embodiments, controller 106 iscoupled to storage device 109 via a high speed peripheral bus such as anSATA bus. Storage device 109 includes an operating system, applicationlevel programs to be executed on one or more processors within thesystem, and other data and/or instructions used to configure variouscomponents or perform the tasks of the present disclosure. Storagedevice 109 may also store data generated by application level programsor by hardware components of the system. For example, network trafficcaptured by capture/offload CLDs 102A may be copied to storage device109 for later retrieval.

Network processor 105 provides software programmable computing that maybe optimized for network applications. Network processor may be aNETLOGIC XLR processor. Network processor 105 is coupled to memory 344,boot flash 326, CPLD 348, and Ethernet transceiver 346. Memory 344 is atangible, computer readable storage medium for storing the instructionsand data of the operating system and application processes running onnetwork processor 105. RAM 332 may be double data rate (DDR3) memory.Boot flash 326 is non-volatile memory storing the operating system imagefor network processor 105. Boot flash 326 may also store applicationsoftware to be executed on network processor 105. CPLD 348 may provideglue logic between network processor 205 and boot flash 326 (e.g.,because the network processor may be capable of interfacing flash memorydirectly). CPLD 348 may also provide reset and power sequencing fornetwork processor 105.

Network processor 105 provides four parallel Ethernet ports, e.g., RGMIIports, for communicating with other devices via the Ethernet protocol.Ethernet transceiver 346, e.g., MARVELL 88E1145 serializes these fourports to provide interoperability with the multiport management switch110. Specifically, in some embodiments, network processor 105 providesfour Reduced Gigabit Media Independent Interface (RGMII) ports, each ofwhich requires twelve pins. The MARVELL 88E1145 transceiver serializesthese ports to reduce the pin count to four pins per port.

Routing FPGA 102B is a configurable logic device configured to routenetwork packets between other devices within the network testing system.Specifically, FPGA 102B is a field programmable gate array and, in someembodiments, is an ALTERA STRATIX 4 device. FPGAs 102 may also be XILINXVIRTEX, ACTEL SMARTFUSION, or ACHRONIX SPEEDSTER parts. Routing FPGA102B may be coupled to tangible computer-readable memory 103B to provideincreased local (to the FPGA) data storage. In some embodiments, memory103B is 8 MB of quad data rate (QDR) static RAM. Static RAM operates ata higher speed than dynamic RAM (e.g., as DDR3 memory) but has a muchlower density.

Offload/capture FPGA 102A is a configurable logic device configured toperform a number of functions as packets are received from externalports 101 or as packets are prepared for transmission on external ports101. Specifically, FPGA 102B is a field programmable gate array and, insome embodiments, is an ALTERA STRATIX 4 device. Offload/capture FPGA102A may be coupled to tangible computer-readable memory 103A to provideincreased local (to the FPGA) data storage. In some embodiments, memory103A is two banks of 16 GB of DDR3 RAM. Memory 103A may be used to storepackets as they are received. Offload/capture FPGA 102A may also becoupled, e.g. via XAUI or SGMII ports to external interfaces 101, whichmay be constructed from physical interfaces 360 and transceivers 362.Physical interfaces 360 convert the XAUI/SGMII data format to a gigabitEthernet signal format. Physical interfaces 360 may be NETLOGIC AEL2006transceivers. Transceivers 362 convert the gigabit Ethernet signalformat into a format suitable for a limited length, direct attachconnection. Transceivers 362 may be SFP+ transceivers for copper offiber optic cabling.

Layer 2/Layer 3 FPGA 102C is a configurable logic device configured togenerate layer 2 or layer 3 egress network traffic. Specifically, FPGA102B is a field programmable gate array and, in some embodiments, is anALTERA STRATIX 4 device.

Management switch 110 is a high-speed Ethernet switch capable of crossconnecting various devices on a single blade or across blades in thenetwork testing system. Management switch 110 may be coupled tonon-volatile memory to provide power-on configuration information.Management switch 110 may be a 1 Gbps Ethernet switch, e.g.,FULCRUM/INTEL FM4000 or BROADCOM BCM5389. In some embodiments,management switch 110 is connected to the following other devices:

-   -   controller 106 (two SGMII connections);    -   each network processor 105 (four SGMII connections);    -   each FPGA 102A, 102B, and 102C (one control connection);    -   backplane 328 (three SGMII connections);    -   external control port 368; and    -   external management port 370.

Serial port access system 366 provides direct data and/or control accessto various system components via controller 106 or an external serialport 372, e.g., a physical RS-232 port on the front of the blade. Serialport access system 366 (illustrated in detail in FIG. 46 and discussedbelow) connects via serial line (illustrated in FIGS. 14A and 14B as anS in a circle) to each of: control processor 106, each network processor105, external serial port 372, and an I2C backplane signaling system374. As discussed below with respect to FIG. 46 I2C backplane signalingsystem 374 may be provided for managing inter-card serial connections,and may include a management microcontroller (or “environmentalcontroller”) 954, I2C connection 958 to backplane 56, and an I2C IOexpander 956. Serial lines may be multipoint low-voltage differentialsignaling (MLVDS).

Alternative System Architecture 100B

FIG. 15 illustrates an alternative testing and simulation architecture100B, according to an example embodiment. Architecture 100B may begenerally similar to architecture 100A shown in FIGS. 4-10, but includesadditional network processors 105 and FPGAs 102. In particular, examplearchitecture 100B includes four network processors 105 and a total of 14FPGAs 102 connected to a management switch 110. In this embodiment, asingle control processor may distribute workloads across two additionalnetwork processors and a total of 14 FPGAs coordinated with a singlehigh-bandwidth Ethernet switch. This embodiment illustrates thescalability of the FPGA pipelining and interconnected FPGA/networkprocessor architecture utilizing Ethernet as a common internalcommunication channel.

FIG. 16 illustrates various sub-systems configured to provide variousfunctions associated with system 16 as discussed herein. For example,control system 450 may include any or all of the following sub-systems:

-   -   An Ethernet-based management system;    -   a distributed DHCP, Addressing and Startup management system;    -   a CLD-based packet routing system;    -   a processor-specific routing system;    -   a CLD pipeline system;    -   a bandwidth management system;    -   a packet capture error tracking system;    -   an efficient packet capture system;    -   a data loopback and capture system;    -   a CLD-based hash function system;    -   multi-key hash tables;    -   a packet assembly subsystem;    -   a packet segmentation offload system;    -   an address compression system;    -   a task management engine;    -   a dynamic latency analysis system;    -   a serial port access system;    -   a USB device initialization system;    -   a USB programming system; and    -   a JTAG programming system.

Each sub-system of control system 450 may include, or have access to,any suitable hardware devices, software, CLD configuration information,and/or firmware for providing the respective functions of thatsub-system, as disclosed herein. The hardware devices, software, CLDconfiguration information, and/or firmware of each respective sub-systemmay be embodied in a single device of system 16, or distributed acrossmultiple devices of 16, as appropriate. The software, CLD configurationinformation, and/or firmware (including any relevant algorithms, code,instructions, or other logic) of each sub-system may be stored in anysuitable tangible storage media of system 16 and may and executable byany processing device of system 16 for performing functions associatedwith that sub-system.

Ethernet Based Management

CLDs in the present disclosure provide specialized functions, butrequire external control and management. In some embodiments of thepresent disclosure, control CPU 106 provides this external control andmanagement for the various CLDs on a board. Control CPU 106 may programany one of the CLDs on the board (e.g., 102A, 102B, 102C, or 123) toconfigure the logic and memory of that CLD. Control CPU 106 may writeinstructions and/or data to a CLD. For example, control CPU 106 may sendinstructions to traffic generating CLD 102C to have that devicegenerating a specified number of network messages in a particular formatwith specified characteristics. In another example, control CPU 106 maysend instructions to capture/offload CLD 102A to read back latencystatistics gathered during a packet capture window.

CLDs are usually managed via a local bus such as a PCI bus. Such anapproach does not scale to large numbers of CLDs and does not facilitateconnectivity between multiple CLDs and multiple CPUs. Some bus designsalso require the payment of licensing fees. The present disclosureprovides a CLD management solution based on the exchange of specializedEthernet packets that can read and write CLD memories (i.e., CLDregisters).

In some embodiments, CLDs in the present disclosure contain embeddedEthernet controllers designed to parse incoming specially formattedpackets as command directives for memory access to be executed. In thisapproach, the CLD directly interprets the incoming packets to make theaccess to internal CLD memory without intervention by an intermediateCPU or microcontroller processing the Ethernet packets. Simultaneousrequests from multiple originating packet sources (e.g., CPUs) aresupported through the use of a command FIFO that queues up incomingrequests. After each command directive is completed by the CLD, aresponse packet is sent back to the originating source CPU containingthe status of the operation.

Three layers of packet definition are used to form the full commanddirective, packet source and destination addressing, the Ethernet typefield, and the register access directive payload. The destination MAC(Media Access Controller) address of each CLD contains the systemmapping scheme for the CLDs while the source MAC contains the identityof the originating CPU. Note that in some embodiments, the MAC addressesof each CLD is only used within the network testing system and are neverused on any external network link. Sub-fields within the destination MACaddress (6 bytes total in length) identify the CLD type, an CLD indexand a board slot ID. The CLD type refers to the function performed bythat particular CLD within the network testing system (i.e., trafficgenerating CLD or capture/offload CLD). A pre-defined Ethernet-Typefield is matched to act as a filter to allow the embedded Ethernetcontroller ignore unwanted network traffic. These 3 fields within thepacket conform to the standard Ethernet fields (IEEE 802.3).

This conformance allows implementation of the network with currentlyavailable interface integrated circuits and Ethernet switches. Ethernetalso requires fewer I/O pins than a bus like PCI, therefore freeing upI/O capacity on the CLD and reducing the trace routing complexity of thecircuit board. Following the MAC addressing and Ethernet type fields aproprietary command format is defined for access directives supported bythe CLD. Some embodiments support instructions for CLD register readsand writes, bulk sequential register reads and writes, and a diagnosticloopback or echo command. Diagnostic loopback or echo commands provide amechanism for instructing a CLD to emulate a network loopback byswapping the source and destination addresses on a packet and insertingthe current timestamp to indicate the time the packet was received.

FIG. 17 illustrates the layout of the Ethernet packets containing CLDcontrol messages according to certain embodiments of the presentdisclosure. The first portion of the packet is the IEEE standard headerfor Ethernet packets, including the destination MAC address, the sourceMAC address, and the Ethernet packet type field. The type field is setto value unused the IEEE standard to avoid conflicts with existingnetwork protocols, especially within the networking stack on the controlCPU. Immediately following the standard header is an access directiveformat including a sequence identifier, a count, a command field, anddata to be used in executing the directive. The sequence number is anidentifier used by the originator of the directive for trackingcompletion and/or timeout of individual directives. The count specifiesthe number of registers accessed by the command and the command fieldspecifies the type of directive.

FIG. 18 illustrates an example register access directive for writingdata to CLD registers, according to certain embodiments of the presentdisclosure. The command field value of 0x0000 indicates a write command.The count field specifies the number of registers to write. The datafield contains a series of addresses and data values to be written.Specifically, the first 32 bits of the data field specify an address.The second 32 bits of the data field specify a value to be written tothe register at the address specified in the first 32 bits of data. Theremaining values in the data field, if any, are arranged in the samepattern: (address, data), (address, data), etc. The response generatedat the completion of the directive is an Ethernet packet with a sourceMAC address of the CLD processing the directive, and a destination MACaddress set to the source MAC address of the packet containing thedirective. The response packet also contains the same Ethernet type,sequence number, and command as the directive packet. The count field ofthe response packet will be set to the number of registers written. Theresponse packet will not contain a data portion.

In certain embodiments, a directive packet can contain only one type ofdirective (e.g., read or write), but can access a large number ofregister addresses within a CLD. In some embodiments, the packet size islimited to the standard maximum transmission unit of 1,500 bytes. Insome embodiments, jumbo frames of 9,000 bytes are supported. By packingmultiple instructions of the same type into a single directive,significant performance enhancement has been observed. In oneconfiguration, startup time of a board was reduced from approximately aminute to approximately five seconds by configuring CLDs over Ethernetinstead of over a PCI bus.

In some embodiments, access directives may be used to access the entirememory space accessible to a CLD. Some CLDs have a flat memory spacewhere a range of addresses corresponds to CLD configuration data,another range of addresses corresponds to internal CLD working memory,and yet another range of addresses corresponds to external memoryconnected to the CLD such as quad data rate static random access memory(QDR) or double data rate synchronous dynamic access memory (DDR).

FIG. 5 illustrates an internal network configuration for certainembodiments of the present disclosure. In FIG. 5, Ethernet switch 110connects to CPU 105 and both NPs 105. In addition, Ethernet switch 110connects to routing CLDs 102B, capture/offload CLDs 102A, and trafficgenerating CLD 102C. In this configuration, any CPU may communicate withany CLD directly using Ethernet packets. Ethernet switch 110 alsoconnects to backplane 56 to extend connectivity to CPUs or CLDs on otherboards. The approach of the present disclosure could also facilitatedirect communication between any of the attached devices including CLDs,network processors, and control processors.

Ethernet switch 110 operates as a layer 2 router with multiple ports.Each port is connected to a device (as discussed in the previousparagraph) or another switch (e.g., through the backplane connection).Ethernet switch 110 maintains a memory associating each port with a listof one or more MAC addresses of the device or devices connected to thatport. Ethernet switch 110 may be implemented as a store and forwarddevice receiving at least part of an incoming Ethernet packet beforemaking a routing decision. The switch examines the destination MACaddress and compares that destination MAC address with entries in theswitch's routing table. If a match is found, the packet will be resentto the assigned port. If a match is not found, the switch may broadcastthe packet to all ports. Upon receipt of a packet, the switch will alsoexamine the source MAC address and compare that address to the switch'srouting table. If the routing table does not have an entry for thesource MAC address, the switch will create an entry associating thesource MAC address with the port on which the packet arrived. In someembodiments, the switch may populates its routing table by sending abroadcast message (i.e., one with a destination address ofFF:FF:FF:FF:FF:FF) to trigger responses from each connected device. Inother embodiments, each device may include an initialization step ofsending an Ethernet message through the switch to announce the device'savailability on the system.

Because Ethernet is a simple, stateless protocol, additional logic isuseful to ensure receipt and proper handling of messages. In someembodiments, each sending device incorporates a state machine to watchfor a response or recognize when a response was not received within apredefined window of time (i.e., a timeout). A response indicating afailure or timeout situation is often reported in a system log. In somesituations, a failure or timeout will cause the state machine to resendthe original message (i.e., retry). In certain embodiments, each processrunning on control processor 106 needing to send instructions to otherdevices via Ethernet may use a shared library to open a raw socket forsending instructions and receiving responses. Multiplexing acrossmultiple processes may be implemented by repurposing the sequence numberfield and setting that field to the process identifier of the requestingprocess. The shared library routines may include filtering mechanisms toensure delivery of responses based on this process identifier (which maybe echoed back by the CLD or network processor when responding to therequest).

In certain embodiments, controller software 132 includes a softwaremodule called an CLD server. The CLD server provides a centralizedmechanism for tracking failures and timeouts of Ethernet commands. TheCLD server may be implemented as an operating system level driver thatimplements a raw socket. This raw socket is configured as a handler forEthernet packets of the type created to implement the CLD controlprotocol. All other Ethernet packets left for handling by thecontroller's networking stack or other raw sockets.

FIG. 19 illustrates an example flow 470 of the life of a register accessdirective, according to certain embodiments of the present disclosure.At step 472, a network processor generates a command for a CLD. Thiscommand could be to generate 10,000 packets containing random data to besent to a network appliance being tested for robustness under heavyload. The network processor generates an Ethernet packet for thedirective with a destination MAC address of the control CPU 106. Thesource MAC address is the MAC address of the network processorgenerating the directive packet. The Ethernet type is set to type usedfor directive packets. The sequence number is set to the currentsequence counter and that counter is incremented. The count field is setto 10,000 and the command field is set to the appropriate command type.The data field contains the destination IP address (or range ofaddresses) and any other parameters needed to specify the trafficgeneration command.

At step 474, the network processor sends the directive packet to controlCPU 106 via switch 110. The directive packet is received by the CLDserver through a raw port on the network driver of the control server.The CLD server creates a record of the directive packet and includes inthat record the current time and at least the source MAC address and thesequence number of the directive packet. The CLD server modifies thedirective packet as follows. The source MAC address is set to the MACaddress of control CPU 106 and the destination MAC address is set to theMAC address of traffic generating CLD 102C. In some embodiments, the CLDserver replaces the sequence number with its own current sequencenumber. In some embodiments, the CLD server may keep a copy of theentire modified directive packet to allow later retransmission.

At step 476, the CLD server transmits the modified directive packet, viaswitch 110, to traffic generating CLD 102C for execution.

At a regular interval, the CLD server examines its records of previouslysent directives to and determines whether any are older than apredetermined age threshold. This might indicate that a response fromthe destination CLD is unlikely due to an error in transmission orexecution of the directive. If any directives are older than thethreshold, then a timeout is recognized at step 478.

In the case of a timeout, the CLD server generates an error message atstep 480 to send to the requesting network processor. In someembodiments, CLD server may resend the directive one or more timesbefore giving up and reporting an error. The CLD server also deletes therecord of the directive packet at this time.

If a response is received prior to a timeout, CLD server removes thedirective packet record and forwards the CLD response packet to theoriginating network processor at step 482. To forward the CLD responsepacket, the CLD server replaces the destination MAC address with the MACaddress of the originating network processor. If the sequence number wasreplaced by the CLD server in step 474, the original sequence number maybe restored. Finally the modified response packet is transmitted, viaswitch 110, to the originating network processor.

While the present disclosure describes the use of Ethernet, othernetworking technologies could be substituted. For example, a copperdistributed data interface (CDDI) ring or concentrator could be used.

Dynamic MAC Address Assignment

In a typical IEEE 802 network, each network endpoint is assigned aunique MAC (Media Access Control) address. Normally the assigned MACaddress is permanent because it is used in layer 2 communications (suchas Ethernet) and unique addressing is a requirement.

As discussed above, network testing system 16 may utilize aconfiguration in which multiple Ethernet-configured devices internallycommunicate with each other over an internal Ethernet interface. In someembodiments, system 16 comprises a chassis 50 with multiple slots 52,and each containing a blade 54 with multiple Ethernet devices, e.g.,CLDs 102, network processors 105, control processor 106, etc.

In some embodiments, the control CPU 106 of each blade 54 is the onlycomponent of system 16 with connectivity to external networks and isthus the public/external Ethernet interface of control CPU 106 is onlycomponent of system 16 that is assigned a globally unique “public” MACaddress. Hardware and software of system 16 dynamically assigns eachother Ethernet device in system 16 (including each network processor105, each CLD 102, and local/internal Ethernet interfaces of control CPU106) a MAC address that is unique within system 16, but need not beglobally unique, as the internal Ethernet network of system 16 does notconnect with external networks. In some embodiments, each of suchEthernet devices is dynamically assigned a unique MAC address based on aset of characteristics regarding that device and its location within theconfiguration of system 16. For example, in some embodiments, eachnetwork processor 105 and CLD in system 16 automatically derives a6-byte MAC address for itself that has the following format:

-   -   1st Byte: fixed (indicates a non-global MAC address).    -   2nd Byte: indicates chip type: e.g., processor, CLD, or other        type of device.    -   3rd Byte: indicates processor type or model, or CLD type or        model: e.g., 20G, 10G, or 1G processor, router CLD,        capture/offload CLD, etc.    -   4th Byte: indicates slot number.    -   5th Byte: indicates processor or CLD number, e.g., to        distinguish between multiple instances of the same type of        processor or CLD on the same card (e.g., two network processors        105 or two capture/offload CLDs 102 a).    -   6th Byte: indicates processor interface (each interface to the        management switch has its own MAC address).

Each CLD (e.g., FPGA 102) derives its own MAC address by reading somestrapping 10 pins on initialization. For example, a four-CLD system mayhave two pins that encode a binary number between 0 and 3. Strappingresistors are connected to these pins for each CLD, and the CLD readsthe value to derive its MAC address. This technique allows systemcontroller 106 to determine all of the encoded information based on theinitial ARP (Address Resolution Protocol) request received from anEthernet device on the internal Ethernet network. This flexibilityallows new blades 54 to be defined that are compatible with existingdevices without causing backwards compatibility problems. For example,if a new blade is designed that is compatible with an old blade, themodel number stays the same. If the new blade adds a new CLD to system16, then the new CLD is simply assigned a different CLD number for theMAC addressing. However, if a new blade is installed in system 16 thatrequires additional functionality on the system controller 106, the newblade may be assigned a new model number. Compatibility with existingblades can thus be preserved.

In addition, the dynamically assigned MAC addresses of Ethernet devicesmay be used by a DHCP server for booting such devices, as discussedbelow in detail.

Each processor may also have an IP address, which may be assigned by theDHCP server based on the MAC address of that device and a set of IPaddress assignment rules.

Distributed DHCP, Addressing and System Start-Up

As discussed above, system 16 may be housed in a chassis 50 thatinterconnects multiple cards 54 via a backplane 56. In some embodiments,all cards 54 boot a single software image. In other embodiments, eachcard 54 runs a different software image, possibly with differentrevisions, in the same chassis 50.

One challenge results from the fact that the cards 54 in chassis 50 arephysically connected to each other via Ethernet over the backplane 56.In addition, some processors in system 16 may obtain their operatingsystem image from other processors across the shared Ethernet usingDHCP. DHCP is a broadcast protocol, such that a request from anyprocessor on any card 54 can be seen from any other card 54. Thus,without an effective measure to prevent it, any processor can boot fromany other processor that replies to its DHCP request quickly enough,including processors on other cards 54 from the requesting processor.This may be problematic in certain embodiments, e.g., embodiments thatsupport hot swapping of cards 54. For example, if a CPU on card 1 bootsfrom a CPU on card 2, and card 2 is subsequently removed from chassis50, CPU 1 may crash.

Thus, in some embodiments (e.g., embodiments that support hot swappingof cards 54), to utilize multiple control processors 105 and drives 109available in a multi-card system 16, as well as to allow for eachcontrol processor 106 to run an independent operating system, whilemaintaining Ethernet connectivity to the backplane 56, system 16 may beconfigured such that local network processors 105 boot from the localcontrol processor 106 using DHCP, NFS (Network File System), and TFTP(Trivial File Transfer Protocol). This task is divided by a specialdynamic configuration for the DHCP server.

First, the network processors 105 and control processor 106 on a card 54determine what physical slot 52 the card 54 is plugged into. The slotnumber is encoded into the MAC address of local network processors 105.The MAC address of each network processor 105 is thus dynamic, but of apredictable format. The DHCP server on the control processor 106configures itself to listen only for requests from network processors105 (and other devices) with the proper slot number encoded in their MACaddresses. Thus, DHCP servers on multiple cards 54 listen for request onthe shared Ethernet, but will only reply to a subset of the possible MACaddresses that are present in system 16. Thus, system 16 may beconfigured such that only one DHCP server responds to a DHCP requestfrom any network processor 105. Each network processor 105 is thusessentially assigned to exactly one DHCP server, the local DHCP server.With this arrangement, each network processor 105 always boots from aprocessor on the same card as that network processor 105 (i.e., a localprocessor). In other embodiments, one or more network processor 105 maybe assigned to the DHCP server on another card, such that networkprocessors 105 may boot from a processor on another card.

A more detailed example of a method of addressing and booting devices insystem 16 is discussed below, with reference to FIGS. 20-22. Asdiscussed above, in a typical Ethernet-based network, each device has aglobally unique MAC address. In some embodiments of network testingsystem 16, the control CPU 106 is the only component of system 16 withconnectivity to external networks and is thus the only component ofsystem 16 that is assigned a globally unique MAC address. For example, aglobally unique MAC address for control CPU may be hard coded into aSPI-4.2 EEPROM 322 (see FIG. 20).

Thus, network processors 105 and CLDs 102 may generate their own MACaddresses according to a suitable algorithm. The MAC address for eachdevice 102, 105, and 106 on a particular card 54 may identify thechassis slot 52 in which that card 54 is located, as well as otheridentifying information. In some embodiments, management switch 110 hasno CPU and runs semi-independently. In particular, management switch 110may have no assigned MAC address, and may rely on control CPU 106 forintelligence.

In some embodiments, network testing system 16 is configured such thatcards 54 can boot and operate independently if desired, and behot-swapped without affecting other the operation of the other cards 54,without the need for additional redundant hardware. Simultaneously,cards 54 can also communicate with each across the backplane 56. Sucharchitecture may improve the scalability and reliability of the system,e.g., in high-slot-count systems. Further, the Ethernet-basedarchitecture of some embodiments may simplify card layout and/or reducecosts.

Cards 54 may be configured to boot up in any suitable manner. FIGS.20-22 illustrate an example boot up process and architecture for a card54 of system 16, according to an example embodiment. In particular, FIG.20 illustrates an example DHCP-based boot management system 290including various components of system 16 involved in a boot up process,FIG. 21 illustrates an example boot-up process for a card 54, and FIG.22 illustrates an example method for generating a configuration file 306during the boot-up process shown in FIG. 21, according to an exampleembodiment.

Referring to FIG. 20, a DHCP-based boot management system 290 mayinclude control CPU 106 connected to a solid-state disk drive 109storing a DHCP server 300, a software driver 302, a configuration script304 configured to generate configuration files 306, an operating system308, a Trivial File Transfer Protocol server (TFTP server) 340, aNetwork Time Protocol (NTP) or Simple Network Time Protocol (SNTP)server 342, and a Network File System (NFS server) 344. Configurationscript 304 may communicate with external hardware via software driver302 and a hardware interface (e.g., JTAG) 310. Controller 106 mayinclude management processor 134, controller software 132, a bootflash320, and an EEPROM 322.

As discussed below, configuration script 304 may be configured to runDHCP server 300, and to automatically and dynamically write newconfiguration files 306 based on the current configuration of system 16,including automatically generating a list of MAC addresses or potentialMAC addresses for various devices for which communications may bemonitored. Configuration script 304 may communicate with system hardwarevia software driver (API) 302 to determine the physical slot 52 in whichthe card 54 is located. Configuration file 306 generated byconfiguration script 304 may include a list of possible valid MACaddresses that may be self-generated by network processors 105 (asdiscussed below) or other offload processors such that DHCP server 300can monitor for communications from network processors 105 on the samecard 54. In some embodiments, configuration file 306 may also listpossible valid MAC addresses for particular devices unable to bootthemselves or particular devices on a card 54 located in a particularslot 52 (e.g., slot 0). Thus, by automatically generating aconfiguration file including a list of relevant MAC addresses,configuration script 304 may eliminate the need to manually compile aconfiguration file or MAC address list.

FIG. 21 illustrates an example method 400 for booting up a card 54 ofsystem 16, according to an example embodiment. The boot-up process mayinvolve management switch 110, controller 106, network processors 105,CLDs 102, and backplane 56.

In general, control CPU 106 boots itself first, then boots managementserver 110, then loads DHCP server 300 and TFTP server 340, NTP server342, and NFS server 344 stored on disk 109. After the control CPU 106finishes loading its servers, each network processor 105 loads itselfand obtains address and other information via a DHCP request andresponse. A more detailed description is provided below.

At step 402, the board 54 is powered. At step 404, management switch 110reads an EEPROM connected to management switch 110, activates localconnections between controller 106, network processors 105, and CLDs102, etc. on card 54, and deactivates backplane connections 328, suchthat all local processors 105 and 106 and CLDs 102 are connected.

In some embodiments, board 54 disables signaling to the backplane 56 (bydeactivating backplane connections 328) and keeps such connectionsdeactivated unless and until board 54 determines a need to communicatedwith another board 54 in system 16. Enabling an Ethernet transceiverwhen there is no receiver on the other side on the backplane 56 causesextra electromagnetic radiation emissions, which may run counter FCCregulations. Thus, disabling backplane signaling may reduce unwantedelectromagnetic radiation emissions, which may place or keep system 16within compliance for certain regulatory standards.

In addition, in one embodiment, each management switch 110 canpotentially connect to three other switches on the backplane 56 (inother embodiments, management switch 110 may connect to more otherswitches). The switch 110 may also provide a function called “loopdetection” that is implemented via a protocol known as “spanning tree.”Loops are typically undesirable in Ethernet systems because a packet mayget caught in the loop, causing a “broadcast storm” condition. Incertain embodiments, the backplane architecture of system 16 is suchthat if every switch 110 comes with its backplane connections enabledand all boards 54 are populated in the system, the switches 110 maydetect a loop configuration and randomly disable ports, depending onwhich port was deemed to be “looped” first by system 16. This may causeboards 54 to become randomly isolated from each other on the backplane56. Thus, by first disabling all backplane connections, and thencarefully only enabling the connections in a manner that prevents a loopcondition from occurring, the possibility of randomly isolating boardsfrom each other may be reduced or eliminated. In other embodiments, thispotential program is addressed by using a different backplane design,e.g., by using a “star” configuration as opposed to a “mesh”configuration, such that the backplane connections may remain enabled.

At step 406, system controller 106 reads bootflash 320 and loads itsoperating system 308 from attached disk drive 109. At step 408, eachnetwork processor 105 reads local bootflash 326 and begins a process ofobtaining an operating system 308 from attached disk drive 109 via DHCPserver 300, by requesting an IP address from DHCP server 300, asdiscussed below. Each network processor 105 can complete the process ofloading an operating system 308 from disk drive 109 after receiving aDHCP response from DHCP server 300, which includes needed informationfor loading the operating system 308, as discussed below. In someembodiments, disk drive 109 stores different operating systems 308 forcontroller 106 and network processors 105. Thus, each processor(controller 106 and individual network processors 105) may retrieve thecorrect operating system 308 for that processor via DHCP server 300.

Bootflash 320 and 326 may contain minimal code sufficient to load therest of the relevant operating system 308 from drive 109. Each networkprocessor 105 on a card 54 automatically derives a MAC address foritself and requests an IP address by sending out a series of DHCPrequests that include the MAC address of that network processor 105. Asdiscussed above, the MAC address derived by each network processor 105may indicate . . . . To derive the slot-identifying MAC address for eachnetwork processor 105, instructions in bootflash 326 may interrogate aComplex Programmable Logic Device (CPLD) 348 to determine which slot 52the card 54 is located in, which may then be incorporated in the MACaddress for the network processor 105. Steps 404, 406, and 408 may occurfully or partially simultaneously.

At step 410, system controller software 132 programs localmicrocontrollers 324 so it can query system status via USB. At step 412,system controller 106 queries hardware slot information to determinewhich slot 52 the card 54 is located. At step 414, system controller 106configures management switch 110 to activate backplane connections 328.Because slots 52 are connected in a mesh fashion by backplane 56, thebackplane connections 328 may be carefully configured to avoid switchloops. For example, in an example 3-slot embodiment: in slot 0, bothbackplane connections 328 are activated; in slot 1, only one backplaneconnection 328 is activated; and in slot 2, the other backplaneconnection 328 is activated.

At step 416, system controller software 132 starts internal NFS server344, TFTP server 340, and NTP server 342 services. At step 418, systemcontroller software 132 queries hardware status, generates a customconfiguration file 304 for the DHCP server 300, and starts DHCP server300. After DHCP server 300 is started, each network processor 105receives a response from DHCP server 300 of the local system controller106 at step 420, in response to the DHCP requests initiated by thatnetwork processor 105 at step 408. The DHCP response to each networkprocessor 105 may include NFS, NTP, TFTP and IP address information, andidentify which operating system 308 to load from drive 109 (e.g., byincluding the path to the correct operating system kernel and filesystemthat the respective network processor 105 should load and run).

At step 422, each network processor 105 configures its network interfacewith the supplied network address information. At step 424, each networkprocessor 105 downloads the relevant OS kernel from drive 109 into itsown memory using TFTP server 340, mounts filesystem via NFS server 344,and synchronizes its time with the clock of the local system controller106 via NTP server 342.

In one embodiment, the NTP time server 342 is modified to “lie” to thenetwork processors 105. Network processors 105 have no “realtime clock”(i.e., they always start up with a fixed date). With the NTP protocol,before an NTP server will give the correct time to a remote client, itmust be reasonably sure that its own time is accurate, determined via“stratum” designation. This normally takes several minutes, whichintroduces an undesirable delay (e.g., the network processor 105 wouldneed to delay boot). Thus, the NTP server immediately advertises itselfas a stratum 1 server to fool the NTP client on the network processors105 to immediately synchronize.

FIG. 22 illustrates an example method 430 for generating a configurationfile 306 during the boot-up process shown in FIG. 21, according to anexample embodiment. At steps 432 and 434, control software 132determines the card type and the slot in which the card 54 is insertedby programming local microcontrollers 324 and querying microcontrollers324 for the blade type and slot ID. At step 436, control software 132determines whether the card is a specific predetermined type of card(e.g., a type of card that includes a local control processor). If so,at step 438, control software 132 activates the configuration script 304to add rules to configuration file 306 that allow booting of localnetwork processors 105 via DHCP server 300. If the card is not thespecific predetermined type of card, control software 132 determineswhether the card is in slot 0 (step 440), and whether any other slot inthe chassis currently contains a different type of card (e.g., a cardthat does not include a local control processor) (step 442). If the cardis in slot 0, and any other slot in the chassis currently contains acard of a type other than the specific predetermined type of card, themethod advances to step 444, in which control software 132 activates theconfiguration script 304 to add rules to configuration file 306 that toallow booting non-local network processors (i.e., NPs in other cards inthe chassis). Control software 132 may determine the number of slots inthe chassis, and add MAC addresses for any processor type (e.g.,particular type of network processor) that does not have a local controlprocessor.

Packet Capture and Routing

CLD-Based Packet Routing

The generalized architecture characteristics of the embodiments of thepresent disclosure enable allows flexible internal routing of receivednetwork messages. However, some applications may require routing rulesto direct traffic matching certain criteria to a specific networkprocessor. For example, in certain embodiments, applications orsituations, when a particular network processor sends a network messageto a device under test it is advantageous that the responsive networkmessage is routed back to the originating network processor, and inparticular to the same core of the originating network processor, e.g.,to maintain thread affinity. As another example, in some embodiments,applications, or situations, all network traffic received on aparticular virtual local area network (VLAN) should be routed to thesame network processor.

These solutions differ from conventional Internet Protocol (IP) routingapproaches, which utilize a table of prefix-based rules In conventionalIP routers, each rule includes an IP address (four bytes in IPv4) and amask indicating which bits of the IP address should be considered whenapplying the rule. The IP router searches the list of rules for eachreceived packet and applies the rule with the longest prefix match. Thisapproach works well for IP routing because rules often apply tosubnetworks defined by a specific number of most significant bits in anIP address. For example, consider a router with the following two ruleprefixes:

a) 128.2.0.0 (255.255.0.0)—all traffic starting with 128.2

b) 128.0.0.0 (255.0.0.0)—all traffic starting with 128

A packet arriving with a destination address of 128.2.1.237 would matchboth rules, but rule “a” would be applied because it matches more bitsof the prefix.

The conventional rule-based approach does not work well for representingrules with ranges. For example a rule applying to IP addresses from128.2.1.2 to 128.2.1.6 would require five separate entries in atraditional routing table including the entries 128.2.1.2, 128.2.1.3,128.2.1.4, 128.2.1.5, and 128.2.1.6 (each with a mask of255.255.255.255).

For certain testing applications, system 16 needs to bind ranges of IPaddresses to a particular processor (e.g., a particular networkprocessor or a particular control CPU). For example, in a networksimulation, each processor may simulate an arbitrary set of hosts on asystem. In certain embodiments, each packet received must arrive at theassigned processor so that the assigned processor can determine whetherresponses were out of sequence, incomplete, or delayed. To achieve thisgoal, routing CLDs 102A may implement a routing protocol optimized forrange matching.

FIG. 23 illustrates portions of an example packet processing and routingsystem 500, according to one embodiment. As shown, packet processing androuting system 500 may include control processor 106, a networkprocessor 105, a routing CLD 102B (e.g., routing FPGA 102B shown inFIGS. 14A-14B), a capture/offload CLD 102A (e.g., capture/offload FPGA102A shown in FIGS. 14A-14B), and test ports 101, and may include aconfiguration register 502, a routing management module 504, a prependmodule 506, a capture logic 520, and a CLD-implemented routing engine508, which may include a static routing module 510, and a dynamicrouting module 512. Each of routing management module 504, prependmodule 506, capture logic 520, and CLD-implemented routing engine 508,including static routing module 510 and dynamic routing module 512 mayinclude any suitable software, firmware, or other logic for providingthe various functionality discussed below. In example FIG. 23,configuration register 502 and prepend module 506 are illustrated asbeing embodied in capture/offload CLD 102A, while routing engine 508,including static routing module 510 and dynamic routing module 512, isillustrated as being embodied in routing CLD 102B. However, it should beclear that each of these modules may be implemented in the other CLD ormay be implemented across both CLD 102A and CLD 102B (e.g., a particularmodule may include certain logic in CLD 102A for providing certainfunctionality associated with that module, and certain other logic inCLD 102B for providing certain other functionality associated with thatmodule).

FIG. 24 is a flowchart illustrating an example method 530 for processingand routing a data packet received by system 16 using example packetprocessing and routing system 500 shown in FIG. 23, according to anexample embodiment. At step 532, a packet (e.g., part of a data streamfrom test network 18) is received at system 16 on a test interface 101and forwarded to capture/offload CLD 102A via a physical interface. Atstep 534, prepend module 506 attaches a prepend header to the receivedpacket. The prepend header may include one or more header fields thatare presently populated, including a timestamp indicting the arrivaltime of the packet, and one or more header fields that may be populatedlater, e.g., a hash value to be subsequently populated by routing module508 in routing CLD 102B, as discussed below. The prepend header isdiscussed in greater detail below, following this description of method530.

At step 536, capture/offload CLD 102A determines whether to capture thepacket in capture buffer 103A, based on capture logic 520. Prior to thestart of the present method, controller 106 may instruct capture logic520 to enable or disable packet capture, e.g., for all incoming packetsor selected incoming packets (e.g., based on specified filters appliedto packet header information). Thus, at step 536, capture/offload CLD102A may determine whether to capture the incoming packet, i.e., store acopy of the packet (including prepend header) in capture buffer 103Abased on the current capture enable/disable setting specified by capturelogic 520 and/or header information of the incoming packet. In oneembodiment, prepend module 506 may include a capture flag in the prependheader at step 534 that indicates (e.g., based on capture logic 520and/or header information of the incoming packet) whether or not tocapture the packet Thus, in such embodiment, step 536 may simply involvechecking for such capture flag in the prepend header.

Based on the decision at step 536, the packet may be copied and storedin capture buffer 103A, as indicated at step 538. The method may thenproceed to the process for routing the packet to a network processor105. Processing and routing system 500 may provide both static (or“basic”) routing and dynamic routing of packets from ports 101 tonetwork processors 105. At step 540, system 500 may determine whether toroute the packet according to a static routing protocol or a dynamicrouting protocol. Routing management module 504 running on controlprocessor 106 may be configured to send instructions to configurationregister 502 on CLDs 102A to select between static and dynamic routingas desired, e.g., manually based on user input or automatically bycontroller 106. Such selection may apply to all incoming packets or toselected incoming packets (e.g., based on specified filters applied topacket header information).

If static (or “basic”) routing is determined at step 540, the packet maybe forwarded to routing CLD 102B, at which static routing module 510 mayapply a static routing algorithm at step 542 to determine a particulardestination processor 105 and physical interface (e.g., a particularSPI-4.2 port bus and/or a particular XAUI port) for forwarding thepacket to the destination processor 105. An example static packetrouting algorithm is discussed below.

Alternatively, if dynamic routing is determined at step 540, the packetmay be forwarded to routing CLD 102B, at which dynamic routing module512 may apply a dynamic routing process at steps 544 through 548 todynamically route the packet to the proper network processor 105, theproper core within that network processor 105, the proper thread groupwithin that core, and the proper thread within that thread group (e.g.,to route the packet to the thread assigned to the conversation in whichthat packet is involved, based on header information of the packet), aswell as providing load balancing across multiple physical interfaces(e.g., multiple SPI4 interfaces) connected to the target networkprocessor 105.

At step 544, dynamic routing module 512 may determine the properdestination network processor 105 and CPU core of that processor basedon dynamic routing algorithms. At step 546, dynamic routing module 512may determine a thread ID associated with the packet being routed. Atstep 548, dynamic routing module 512 may determine select a physicalinterface (e.g., a particular SPI4 interface) over which to route thepacket to the destination network processor 105, e.g., to provide loadbalancing across multiple physical interfaces. Each of these steps ofthe dynamic routing process, 544, 546, and 548, is discussed below ingreater detail. It should also be noted that one or more of theseaspects of the dynamic routing process may be incorporated into thestatic routing process, depending on the particular embodiment and/oroperational situation. For example, in some embodiments, static routingmay incorporate the thread ID determination of step 546 in order toroute the packet to a particular thread corresponding to that packet.

Once the static or dynamic routing determinations are made as discussedabove, routing CLD 102B may then route packet to the determined networkprocessor 105 over the determined routing path (e.g., physicalinterface(s)) at step 550. At step 552, the network processor 105receives the packet and places the packet in the proper thread queuebased on the thread ID determined at step 546. At step 554, the networkprocessor 105 may then process the packet as desired, e.g., using anyapplication-level processing. Various aspects of the routing method 530are now discussed in further detail.

Prepend Header

In some embodiments, once the key has been obtained for an ingresspacket, routing engine 508 may prepend a destination specific header tothe packet. Likewise, every packet generated by control processor 106 ornetwork processor 105 for transmission by interface 101 includes aprepend header that will be stripped off by capture/offload CLD 102Aprior to final transmission. These prepend headers may be used to routethis traffic internally n system 16.

The prepend header added by capture/offload CLD 102A to ingress packetsarriving at interface 101 for delivery to a network processor maycontain the following information, according to certain embodiments ofthe present disclosure:

struct np_extport_ingress_hdr {   uint32_t timestamp;   uint32_tphysical_interface:3;   uint32_t thread_id:5;   uint32_t l3_offset:8;  uint32_t l4_offset:8;   uint32_t flags:8;   uint32_t hash;   uint32_tunused; } _attribute_((_packed_))

The np_extport_ingress_hdr structure defines the prepend fields set onall packets arriving from an external port to be processed by a networkprocessor, according to certain embodiments of the present disclosure.The timestamp field may be set to the time of receipt by thecapture/offload CLD 102A receiving the packet from interface 101. Thistimestamp may be used to determine all necessary and useful statisticsrelating to timing as it stops the clock prior to any internal routingor transmission delays between components within the network testingsystem. The physical_interface field (which may be set by routing engine508) contains information sufficient to uniquely identify the physicalport on which the packet was originally received. The thread_id fieldcontains information sufficient to uniquely identify the software threadon the network processor that will process this incoming packet.

As described elsewhere in this specification, maintaining ordering andassigning packets to thread groups ensures that the testing applicationhas complete visibility into all of the packets in a given testscenario. The L3 and L4 offset fields indicate the location within theoriginal packet of the OSI layer three and four packet headers. In someembodiments, these offset fields may be is determined by capture/offloadCLDs 102A and stored for later use. Header offsets may be time-consumingto determine due to the possible presence of variable-length optionfields and additional embedded protocol layers. Because the headeroffsets must be determined in order to perform other functions (e.g.,checksum verification described below), this information may efficientlybe stored in the prepend header for future reference. For instance,parsing VLAN tags can be time-consuming because there may be manydifferent values that may be used for VLAN tag identification, andbecause VLAN headers may be stored on unaligned boundaries. However, ifthe capture/offload CLD 102A indicates that the L3 header is at a 14byte offset, this fact may immediately indicate the lack of VLAN tags.In that case, routing engine 508 and/or network processor 105 may skipVLAN parsing altogether. In another instance, if parsing L3 headers(IPv4 and IPv6) can be slowed by the presence of option headers, whichare of variable length. By looking at the L4 header byte offset, networkprocessor 105 can immediately determine whether options are present andmay skip attempts to parse those options if they are not present.

The flags field indicates additional information about the packet asreceived. In some embodiments, flags may indicate whether the certainchecksum values were correct, indicating that the data was likelytransferred without corruption. For example, flags may indicate whetherlayer 2, 3, or 4 checksums were valid or whether an IPv6 tunnel checksumis valid. The hash field is the hash value determined by capture/offloadCLDs 102A and stored for later use.

The prepend header for packets generated by a network processor fortransmission via interface 101 may contain the following information,according to certain embodiments of the present disclosure:

struct np_extport_egress_hdr {   uint32_t unused;   uint32_tphysical_interface:3;   uint32_t unused2:13;   uint32_ttimestamp_word_offset:8;   uint32_t flags:8;   uint32_t unused3[2]; }_attribute_((_packed_));

The np_extport_egress_hdr structure defines the prepend fields set onall packets generated by a network processor to be sent on an externalport to be processed by a network processor, according to certainembodiments of the present disclosure. The physical_interface fieldcontains information sufficient to identify the specific physicalinterface on which the packet was received. The timestamp_word_offsetfield indicates the location within the packet of the timestamp fieldfor efficient access by capture/offload CLD 102A.

The prepend header for packets arriving via interface 101 for deliveryto a control processor may contain the following information, accordingto certain embodiments of the present disclosure:

struct bps_extport_ingress_hdr {   uint32_t timestamp;   uint8_t intf;  uint8_t l3_offset;   uint8_t l4_offset;   uint8_t flags; // signals toprocessors the status of checksums   uint32_t hash;   uint16_t ethtype;// used to fool Ethernet MAC (0x800)   uint16_t thread_id ; // used forrouting packets to a particular // core/thread within a processor };

The ethtype field is included in the prepend header and set to 0x800(e.g., the value for Internet Protocol, Version 4 or IPv4) for ingressand egress traffic, though it ignored by the CLD and network processorhardware/software. This type value used to fool the Ethernet interfacechipset (e.g., the INTEL 82599 Ethernet MAC or other suitable device)interfaced with the control processor into believing the traffic isregular IP over Ethernet when the system is actually using the area as aprepend header. Because this is a point-to-point link and because thedevices on each end of the communication channel are operating in a rawmode or promiscuous mode, the prepend header may be handled properly onboth ends without confusing a traditional networking stack. If theethtype field were set to any value less than 0x600, the value would betreated as length instead under IEEE Standard 802.3x-1997.

The fields of the ingress prepend header for packets arriving on anexternal port and transmitted to the control processor are listed in thestructure named bps export ingress hdr. The timestamp field is set tothe time of receipt by the capture/offload CLD 102A receiving the packetfrom interface 101. The intf field specifies the specific interface 101on which the ingress packet arrived. The L3 and L4 offset fieldsindicate the location within the original packet of the OSI layer threeand four packet headers. The flags field indicates additionalinformation about the packet as received. In some embodiments, flags mayindicate whether the certain checksum values were correct, indicatingthat the data was likely transferred without corruption. For example,flags may indicate whether layer 2, 3, or 4 checksums were valid orwhether an IPv6 tunnel checksum is valid. The hash field is the hashvalue determined by capture/offload CLDs 102A and stored for later use.The thread_id field contains information sufficient to uniquely identifythe software thread on the network processor that will process thisincoming packet.

The prepend header for packets generated by a control processor fortransmission via interface 101 may contain the following information,according to certain embodiments of the present disclosure:

struct bps_extport_egress_hdr {   uint16_t l3_tunnel_offset;   uint16_ttcp_mss; // signals to tcp segmentation offload engine the MSS   uint8_tunused;   uint8_t intf; // test interface to send a packet on   uint8_ttimestamp_word_offset; // signals where to insert the   timestamp  uint8_t flags;   uint32_t unused1;   uint16_t ethtype; // used to foolEthernet MAC (0x800)   uint16_t unused2; };

The fields of the engress prepend header for packets generated by acontrol processor for transmission via an external port are listed inthe structure named bps_export_egress_hdr. The field 13_tunnel_offsetidentifies the start of the layer 3 tunneled packet header within thepacket. The field tcp_mss is a maximum segment size value for use by theTCP segmentation offload processing logic in capture/offload CLD 102A.The intf field specifies the specific interface port 101 that shouldtransmit the packet. The field timestamp_word_offset specifies thelocation in the packet where the capture/offload CLD 102A should insertthe timestamp just prior to transmitting the packet.

The flags field may be used to trigger optional functionality to beperformed by, e.g., capture/offload CLD 102A prior to transmission ofegress packets. For example, flag bits may be used to instructcapture/offload CLD 102A to generate and set checksums for the IPheader, L4 header (e.g., TCP, UDP, or ICMP), and/or a tunnel header. Inanother example, a flag bit may instruct capture/offload CLD 102A toinsert a timestamp at a location specified by timestamp_word_offset. Inyet another example, a flag bit may be used to instruct capture/offloadCLD 102A to perform TCP segmentation using the tcp_mss value as amaximum segment size.

In some embodiments, the prepend header is encapsulated in anotherEthernet header (so a packet would structure be (Ethernet header→prependheader→real Ethernet header). Such embodiments add an additional 14bytes per-packet in overhead to the communication process versustricking the MAC using 0x800 as the ethtype value.

Static (“Basic”) Packet Routing

Basic packet routing mode statically binds a port to a particular to adestination processor and bus/port. In some embodiments, configurationregister 502 takes on the following meaning in the basic packet routingmode:

Configuration Register, Address 0x000F_0000:   bits [1:0] = Destinationfor port 0     00 = NP 0     01 = NP 1     10 = X86     11 = Invalid,packets will get dropped.   bits [6:2] = Invalid in static routing mode  bit [7] = destination bus for port 0     0 = SPI 0/XAUI 0     1 = SPI1/XAUI 1   bit [8] = invalid in static routing mode   bit [9] = enableCAM on port 0     0 = static routing mode     1 = dynamic routing/camrouting mode   bits [17:16] = Destination for port 1     00 = NP 0    01 = NP 1     10 = X86     11 = Invalid, packets will get dropped.  bits [22:18] = Invalid in static routing mode   bit [23] = destinationbus for port 1     0 = SPI 0/XAUI 0     1 = SPI 1/XAUI 1   bit [24] =invalid in static routing mode   bit [25] = enable CAM on port 1     0 =static routing mode     1 = dynamic routing/cam routing mode

Dynamic Routing

Packet processing and routing system 500 may provide dynamic packetrouting in any suitable manner. For example, with reference to steps544-548 of method 530 discussed above, dynamic routing module 512 maydetermine the proper destination network processor 105 and CPU core ofthat processor based on dynamic routing algorithms, determine a threadID associated with the packet being routed, and select a physicalinterface (e.g., a particular SPI4 interface) over which to route thepacket to the destination network processor 105, e.g., to provide loadbalancing across multiple physical interfaces.

In certain embodiments, dynamic routing module 512 is configured todetermine ingress routing based on arbitrary IPv4 and IPv6 destinationaddress ranges and VLAN ranges. Routing module 508 examines each ingresspacket and generates a destination processor 105 and a thread groupidentifier associated with that processor. Thread groups are a logicalconcept on the network processors that each contain some number ofsoftware threads (i.e., multi-processing execution contexts). The secondrouting stage calculates a hash value (e.g., jhash value) based onparticular header information in each ingress packet: namely, the sourceIP address, destination IP address, source port, and destination port.This hash value is used to determine which thread within the threadgroup determined by the CAM lookup to route the packet. In someembodiments, a predefined selected bit (e.g., a bit predetermined in naysuitable manner as the least significant bit (LSB)) of the hash is alsoused to determine which of multiple physical interfaces on the CPU (ie:SPI 0 or 1, or XAUI 0 or 1) to route the packet, e.g., to provide loadbalancing across the multiple physical interfaces.

The Content Addressable Memory (CAM) Lookup

FIG. 25 illustrates dynamic routing determination 570, according tocertain embodiments of the present disclosure. At step 572, dynamicrouting module 512 may extract destination IP address and VLANidentifier from the ingress packet to be routed. This extraction processmay require routing CLD 102B to reparse the L3 headers of the ingresspacket if the IP destination address and VLAN identifier were not storedin the prepend header by capture/offload CLD 102A.

At step 574, dynamic routing module 512 may perform a lookup into theVLAN table indexed by the VLAN identifier extracted from the packet tobe routed. At step 576, dynamic routing module 512 may search theexception table for an entry matching the destination IP address of theingress packet, or may fall back on a VLAN or system-wide default.

This method may be better understood in the context of certain datastructures referenced above. Routing entries are stored in IP addressranges on a per-VLAN basis. The CAM is made up of a 4k×32 VLAN table(e.g., one entry per possible VLAN value), a 16k×256 Exception table,and a 16k×32 key table. The VLAN table may indicate the defaultdestination for that VLAN, and may contain the location in the exceptiontable that contains IP ranges associated with that VLAN. In someembodiments, the VLAN table may include start and end indices into theexception table to allow overlap and sharing of exception table entriesbetween VLAN values. Each of these tables may be setup or modified byrouting management module 504. Additional information about these threerouting tables is included as follows, according to certain embodimentsof the present disclosure.

-   -   The VLAN table may be located at the following base address        (e.g., in the address space of routing CLD 102B):

port 0 = 0x0030_0000-0x0030_0FFF port 1 = 0x0050_0000-0x0050_0FFF

-   -   The Exception table may be located at the following base address        (e.g., in the address space of routing CLD 102B):

port 0 = 0x0020_0000-0x0028_FFFF port 1 = 0x0040_0000-0x0048_FFFF

-   -   Configuration Register, Address 0x000F_(—)0000 (e.g., in the        address space of routing CLD 102B):

bits [5:0] = Default key for port 0 bit [8] = enable ipv6 for port 0 bit[9] = enable CAM on port 0 bits [21:16] = Default key for port 1 bit[24] = enable ipv6 for port 1 bit [25] = enable CAM on port 1

-   -   In certain embodiments, each entry of the VLAN table may be a 32        bit word formatted as follows:

bits [14:0]: address of the first IP range entry in the Exception tablefor     this VLAN bits[15]: VLAN valid. This bit must be set to 1 forthis VLAN entry to     be considered valid bits[21:16]: Number ofexceptions for this VLAN bits[30:24]: Default destination. Use thisvalue if no range is matched in     the exception table. bits[31]:unused

In certain embodiments, address bits [13:2] of the VLAN table are theVLAN identifier. So, to configure VLAN 12′h2783 for port 0, you wouldwrite to location 0x309E0C. Because entries in the VLAN table have astart address into the Exception Table and a count (e.g., bits[21:16]),it is possible to have VLAN entries with overlapping rules or one VLANentry may reference a subset of the exception table referenced byanother VLAN entry.

The exception table may contain all of the IP ranges for each VLAN. Anygiven entry in the exception table can contain 1 IPv6 exception or up to4 IPv4 exceptions. IPv6 and IPv4 cannot be mixed in a single entry,however there is no restriction on mixing IPv6 entries with IPv4entries.

Exception Base Address:

port 0 = 0x20_0000 port 1 = 0x40_0000

TABLE 3 Exception Format bits bits bits bits Bits Bits Bits Bits offset[255:224] [223:192] [191:160] [159:128] [127:96] [95:64] [63:0] [31:0]Base + IPV4 IPV4 IPV4 IPV4 IPV4 IPV4 IPV4 IPV4 (Row * Range 3 Range 3Range 2 Range 2 Range 1 Range 1 Range 0 Range 0 32) Upper lower Upperlower Upper lower Upper lower Address address Address address Addressaddress Address address IPV6 Range 0 Upper IPV6 Range 0 Lower

Key Base Address:

port 0 = 0x28_0000 port 1 = 0x48_0000

TABLE 4 Key Table Format Bits Bits Bits Bit offset bit[31] [30:24][22:16] [14:8] [6:0] Base + IPV6 Enable IPV4 IPV4 IPV4 IPV4 (Row*4) 0 =Row is 4 Range Range Range Range IPV4 Ranges 3 Key 2 Key 1 Key 0 Key 1 =Row is 1 Unused in IPV6 IPV6 IPV6 Range Range 0 Key

In certain embodiments, each entry of the VLAN table has the followingformat:

Exception Table: bits[255:224]: Range 3 upper address bits[223:192]:Range 3 lower address bits[191:160]: Range 2 upper addressbits[159:128]: Range 2 lower address bits[127:96]: Range 1 upper addressbits[95:64]: Range 1 lower address bits[63:32]: Range 0 upper addressbits[31:0]: Range 0 lower address Key Table: bit [31]: 0 = IPv4, 1 =IPv6. Bits[30:24]: Key for range 3 bits[22:16]: Key for range 2bits[14:8]: Key for range 1 bits[6:0]: Key for range 0

When a match is found for the destination IP address (it falls within arange defined in the exception table), the key for that entry isreturned. If no match is found for that entry, the default key for thatVLAN is returned. If there is no match for that VLAN, then the defaultkey for the test interface is returned. In some embodiments, the formatof the key is as follows:

key bits [1:0] = Destination processor     00 = NP0     01 = NP1     10= X86     11 = UNUSED key bits [5:2] = Processor thread group.

Flow Affinity

Many network analysis mechanisms require knowledge of the order in whichpackets arrived at system 16. However, with the significant parallelismpresent in system 16 (e.g., multiple processors and multiple cores perprocessor), a mechanism is needed to ensure packet ordering. Oneapproach employed is a method called “flow affinity.” Under this method,packets for a given network traffic flow should always be received andprocessed by the same CPU thread. Otherwise, packets may be processedout of order as a flow ping-pongs between CPU threads, reducingperformance as well as causing false-positive detection of packet lossfor network performance mechanisms like TCP fast-retransmit. Therudimentary hardware support for flow affinity provided by networkprocessor 105 is simply not sufficiently flexible to account for all thetypes of traffic processed by system 16. The present disclosure presentsa flexible flow affinity solution through a flow binding algorithmimplemented in a CLD (e.g., routing CLD 102B).

FIG. 25 illustrates the flow affinity determination 580, according tocertain embodiments of the present disclosure. At step 582, routingmodule 508 parses each ingress packet to extract flow information, forexample the 4-tuple of: destination IP address, source IP address,destination port, and source port of the ingress packet. This 4-tupledefines a flow. In some embodiments, the flow identifying informationmay be numerically sorted to ensure the same 4-tuple for packets sent inboth directions, especially where system 16 is operating as a “bump inthe line” between two devices under observation. In other embodiments,source and destination information may be swapped for packets receivedon a specific external interface port to achieve a similar result. Atstep 584, jhash module 516 calculates a hash value on the flowidentification information. In some embodiments, the extraction and hashsteps are performed elsewhere, e.g., offload/capture CLD 102A and thehash value is stored in the prepend header for use by flow affinitydetermination 580.

At step 586, routing module 508 looks up in Table 5 the number ofthreads value and starting thread value corresponding to the previouslydetermined (e.g., at step 576) thread group and processor identifier forthe packet. In some embodiments, each processor may have up to 16 threadgroups. In other embodiments, each processor may have up to 32 threadgroups. A thread group may have multiple threads associated with it. Therouting management module 504 may configure the thread associations,e.g., by modifying Table 5 on routing CLDs 102B.

TABLE 5 Thread Group NP0: NP1: X86: Bits[12:8] Bits [4:0] 0 0xF_02000xF_0280 0xF_0300 Num threads Starting Thread 1 0xF_0204 0xF_02840xF_0304 Num threads Starting Thread 2 0xF_0208 0xF_0288 0xF_0308 Numthreads Starting Thread 3 0xF_020C 0xF_028C 0xF_030C Num threadsStarting Thread 4 0xF_0210 0xF_0290 0xF_0310 Num threads Starting Thread5 0xF_0214 0xF_0294 0xF_0314 Num threads Starting Thread 6 0xF_02180xF_0298 0xF_0318 Num threads Starting Thread 7 0xF_021C 0xF_029C0xF_031C Num threads Starting Thread 8 0xF_0220 0xF_02A0 0xF_0320 Numthreads Starting Thread 9 0xF_0224 0xF_02A4 0xF_0324 Num threadsStarting Thread 10 0xF_0228 0xF_02A8 0xF_0328 Num threads StartingThread 11 0xF_022C 0xF_02AC 0xF_032C Num threads Starting Thread 120xF_0230 0xF_02B0 0xF_0330 Num threads Starting Thread 13 0xF_02340xF_02B4 0xF_0334 Num threads Starting Thread 14 0xF_0238 0xF_02B80xF_0338 Num threads Starting Thread 15 0xF_023C 0xF_02BC 0xF_033C Numthreads Starting Thread 16 0xF_0240 0xF_02C0 0xF_0340 Num threadsStarting Thread 17 0xF_0244 0xF_02C4 0xF_0344 Num threads StartingThread 18 0xF_0248 0xF_02C8 0xF_0348 Num threads Starting Thread 190xF_024C 0xF_02CC 0xF_034C Num threads Starting Thread 20 0xF_02500xF_02D0 0xF_0350 Num threads Starting Thread 21 0xF_0254 0xF_02D40xF_0354 Num threads Starting Thread 22 0xF_0258 0xF_02D8 0xF_0358 Numthreads Starting Thread 23 0xF_025C 0xF_02DC 0xF_035C Num threadsStarting Thread 24 0xF_0260 0xF_02E0 0xF_0360 Num threads StartingThread 25 0xF_0264 0xF_02E4 0xF_0364 Num threads Starting Thread 260xF_0268 0xF_02E8 0xF_0368 Num threads Starting Thread 27 0xF_026C0xF_02EC 0xF_036C Num threads Starting Thread 28 0xF_027D 0xF_02F00xF_0370 Num threads Starting Thread 29 0xF_0274 0xF_02F4 0xF_0374 Numthreads Starting Thread 30 0xF_0278 0xF_02F8 0xF_0378 Num threadsStarting Thread 31 0xF_027C 0xF_02FC 0xF_037C Num threads StartingThread

At step 588, routing module 508 may calculate the thread identifierbased on the following formula:

thread[4:0]=“Starting Thread”+(hash value MOD “Num threads”)

At step 590, routing module 508 may update the packet's prepend headerto include the thread identifier for subsequent use by the networkprocessor.

Hash Function

Dynamic routing module 512 may perform a hash function in parallel with(or alternatively, before or after) the CAM lookup and/or other aspectsof the dynamic routing process. Dynamic routing module 512 may extractheader information from each ingress packet and calculates a hash valuefrom such header information using the jhash algorithm. In a particularembodiment, dynamic routing module 512 extracts a 12-byte“4-tuple”—namely, source IP, destination IP, source port, and destport—from the IP header and UDP header of each ingress packet, andapplies a jhash algoritihm 516 to calculate a 32-bit jhash value fromsuch 4-tuple. Dynamic routing module 512 may parse and calculate thehash value each packet at line rate in the FPGAs, which may thereby freeup processor cycles in the network processors 105. For example, dynamicrouting module 512 may embed the calculated hash value is into theprepend header of each packet so that the network processors 105 canmake use of the hash without having to parse the packet or calculate thehash. Dynamic routing module 512 can then use the embedded jhash valuesfor packet routing and load balancing as discussed herein.

As discussed herein, system 16 may utilize the jhash function written byBob Jenkins (see http://burtleburtle.net/bob/c/lookup3.c) for variousfunctions. As shown, the jhash function may be implemented by CLDs,e.g., FPGAs 102A and/or 102B of the example embodiment of FIGS. 14A-14B.For example, capture/offload FPGAs 102A or routing FPGA 102B may applythe jhash function to header information of incoming packets asdiscussed above, which may allow increased throughput through system 16as compared to an arrangement in which the hash functions areimplemented by network processors 105 or other CPUs.

In some embodiments, dynamic routing module 512 pre-processes the4-tuple information before applying the hash function such that allcommunications of a particular two-way communication flow—in bothdirections—receive the same hash value, and are thus routed to the sameprocessor core in order to provide flow affinity. Packets flowing indifferent directions in the same communication flow will have oppositesource port and destination port data, which would lead to differenthash values (and thus potentially different routing destinations) forthe two sides of a particular conversation. Thus, to avoid this result,in one embodiment dynamic routing module 512 05 utilizes a tupleordering algorithm 518 that orders the four items of the 4-tuple innumerical order (or at least orders the source port and destinationport) before applying the hash algorithm, such that the ordered tuple towhich the hash function is applied is the same for both sides of theconversation. This technique may be useful for particular applications,e.g., in “bump in the wire” configurations where it is desired tomonitor both sides of a conversation (e.g., for simulating or testing afirewall).

Further, dynamic routing module 512 may use jhash value to determinewhich of multiple physical interfaces (e.g., multiple SPI-4.2 interfacesor multiple XAUI interfaces) to route each packet, e.g., for loadbalancing across such physical interfaces. For example, a predefinedselected bit (e.g., a bit predetermined in any suitable manner as theleast significant bit (LSB)) of the hash may be used to determine whichphysical interfaces on the CPU (e.g., SPI-4.2 port 0 or 1, or XAUI 0or 1) to route the packet. In an example embodiment, bit 10 of the jhashvalue was selected to determine the port to route each packet. Thus, inan example that includes two SPI interfaces (SPI-4.2 ports 0 and 1)between routing CLD 102B and network processor 105, if hash[10]==0 for aparticular packet, routing CLD 102B will forward the packet on SPI-4.2port 0, and if hash[10]==1, it will send the packet on SPI-4.2 port 1.Using the hash value in this manner may provide a deterministicmechanism for substantially evenly distributing traffic over two or moreparallel interconnections (e.g., less than 1% difference in trafficdistribution over each of the different interconnections) due to thesubstantially random nature of the predefined selected hash value bit.

The process discussed above may ensure that all packets of the samecommunication flow (and regardless of their direction in the flow) areforwarded not only to the same network processor 105, but to thatprocessor 105 via the same physical serial interface (e.g., the sameSPI-4.2 port), which may ensure that packets of the same communicationflow are delivered to the network processor 105 in the correct order,due to the serial nature of such interface.

Processor-Specific Routing

While many components provide channelized interconnections such as theSPI4 interconnections on the FPGAs, general purpose CPUs often do not.General purpose CPUs are designed to operate more as controllers ofspecialized devices rather than peers in a network of other processors.FIGS. 14A and 14B illustrate an approach to providing a channelizedinterconnection between the general purpose CPU of controller 106 andthe routing CLDs 102B (shown as FPGAs), according to some embodiments ofthe present disclosure.

In FIG. 14A, INTEL XEON processor (labeled Intel Jasper Forrest) isconfigured as control processor 106. This processor is a quad core, x86compatible processor with a Peripheral Component Interconnect Express(PCIe) interconnection to two INTEL 82599 dual channel 10 Gbps Ethernetmedium access controllers (MACs). Rather than operating as traditionalnetwork connections, these components are configured to providechannelized data over four 10 Gbps connections. In particular, directconnections are provided between one of the INTEL 82599 MACs and the twoRouting FPGAs 102B.

In this configuration, the prepend header (discussed above) is used tosignal to the MAC that the packet should be passed along as an IPpacket. The control processor has a raw packet driver that automaticallyadds and strips the prepend header to allow software processing ofstandard Ethernet packets.

As with the two SPI4 ports on the routing CLDs 102B, ingress traffic tothe control processor should be load balanced across the two 10 GbpsEthernet channels connecting the routing CLDs 102B and the INTEL 82599.The load balancing may operate in the same manner as that describedabove in the context of the SPI4 ports, based on a hash value. However,the routing process is more complicated. An ingress packet arriving atthe routing CLD 102B illustrated in FIG. 14A will either be routedthrough the 10 Gbps Ethernet (e.g., XAUI) connection directly to theINTEL 82599 MAC or will be routed through the routing CLD 102Billustrated in FIG. 14B (e.g., via interconnection 120). In the latterscenario, routing CLD 102B illustrated in FIG. 14B will then route thepacket through that CLD's 10 Gbps Ethernet (e.g., XAUI) connectiondirectly to the INTEL 82599 MAC.

CLD Pipeline

In certain applications, the complexity of logic to be offloaded from aprocessors to a CLD becomes too great to efficiently implement in asingle CLD. Internal device congestion prevents the device fromprocessing traffic at line rates. Further, as the device utilizationincreases, development time increases much faster than a linear fashionas development tools employ more sophisticated layout techniques andspend more time optimizing. Traditional design approaches suggestsolving this problem by selecting a more complex and capable CLD partthat will provide excess capacity. Fewer components often reducesoverall design and manufacturing costs even if more complex parts areindividually more expensive.

In contrast, certain embodiments of the present invention take adifferent approach and span functionality across multiple CLDs in acareful deintegration of functionality. This deintegration is possiblewith careful separation of functions and through the use of low latency,high-throughput interconnections between CLDs. In some embodiments, aproprietary bus (e.g., the ALTERA SERIALLITE bus) is used to connect twoor more compatible CLD devices to communicate with latencies andthroughput approximating that of each device's internal I/O channels.This approach is referred to herein as pipelining of CLD functionality.Pipelining enables independent design and development of each module andthe increased availability of I/O pins at the cost of additionalprocessing latency. However, certain applications are not sensitive toincreased latency. Many network testing applications fall into thiscategory where negative effects of processing latency can be effectivelyneutralized by time stamping packets as they arrive.

In the embodiments illustrated by FIG. 4, CLD functionality isdistributed across three CLDs. In these embodiments, egress networktraffic either flows through routing CLD 102B and capture/offload CLD102A or through traffic generating CLD 102C and capture/offload CLD102A. Likewise, ingress network traffic flows through capture/offloadCLD 102A and routing CLD 102B. The functions assigned to each of thesedevices is described elsewhere in this disclosure.

Bandwidth Management

In certain embodiments, each network processor has a theoreticalaggregate network connectivity of 22 Gbps. However, this connectivity issplit between two 11 Gbps SPI4 interfaces (e.g., interfaces 122). Themethod of distributing traffic across the two interfaces is a criticaldesign consideration as uneven distribution would result in asignificant reduction in the achieved aggregate throughput. For example,statically assigning a physical network interface (e.g., interface 101)to an SPI4 interface may not allow a single network processor to fullysaturate a physical interface with generated network traffic. In anotherexample, in some applications it is desirable to have a single networkprocessor saturate two physical network interfaces. The user should notneed to worry about internal device topologies in configuring such anapplication. Another core design constraint is the need to maintainpacket ordering for many applications.

In some embodiments, software on a network processor assigns SPI4interfaces to processor cores in the network processor such that allegress packets are sent on the assigned SPI4 interface. In someembodiments, processor cores with an odd number send data on SPI4-1while those with an even number send data on SPI4-0. A simple bit maskoperation can be used to implement this approach: SPI4 Interface—CORE_ID& 0x1. This approach could be scaled to processors with additional SPI4ports using a modulus function.

In certain embodiments, ingress packets are routed through specific SPI4interfaces based on the output of an appropriate hashing algorithm, forexample the jhash algorithm described below. In some embodiments, thesource and destination addresses of the ingress packet are input intothe hashing algorithm.

In situations where the hashing algorithm varies based on the order ofthe input, it may be desirable to route packets between the same twohosts to the same interface on the network processor. For example, thenetwork testing device may be configured to quietly observe networktraffic between two devices in a “bump in the line” configuration. Inthis scenario, the routing CLD may first numerically sort the source anddestination address (along with any other values input into the hashfunction) to ensure that the same hash value is generated regardless ofwhich direction the network traffic is flowing.

Packet Capture Error Tracking

In certain embodiments, offload/capture CLDs 102A are configured tocapture and store packets received on interfaces 101 in capture memory103A. Packets may be captured to keep a verbatim record of allcommunications for later analysis or direct retrieval. Captured packetsmay be recorded in a standard format, e.g., the PCAP format, or withsufficient information to enable later export to a standard format.

With modern data rates on the order of 10 Gbps, packet capture mayconsume a significant amount of memory in a very short window of time.In certain embodiments, the packet capture facility of offload/captureCLDs 102A may be configurable to conserve memory and focus resources. Insome embodiments, the packet capture facility may capture a limitedwindow of all received packets, e.g., through the use of a circularcapture memory described below. In certain embodiments, the packetcapture facility may incorporate triggers to start and stop the captureprocess based on certain data characteristics of the received packets.

In some embodiments, offload/capture CLDs 102A verify one or morechecksum values on each ingress packet. The result of that verificationmay be used to set one or more flags in the prepend header, as discussedelsewhere in this disclosure. Examples of checksums include the layer 2Ethernet checksum, layer 3 IP checksum, layer 4 TCP checksum, and IPv6tunneling checksum. Erroneous packets may be captured in order toisolate and diagnose the source of erroneous traffic.

In some embodiments, offload/capture CLDs 102A may apply a set of rulesagainst each ingress packet. For example, packet sizes may be monitoredto look for abnormal distributions of large or small packets. Anabnormal number of minimum size or maximum size packets may signalerroneous data or a denial of service attack. Packet types may bemonitored to look for abnormal distributions of layer 4 traffic. Forexample, a high percentage of TCP connection setup traffic may indicatea possible denial of service attack. In another example, a particularpacket type, e.g, an address resolution protocol packet or a TCPconnection setup packet, may trigger packet capture in order to analyzeand/or record logical events.

In some embodiments, offload/capture CLDs 102A may include a statemachine to enable capture of a set of packets based on a event trigger.This state machine may begin capturing packets when triggered by one ormore rules described above. The state machine may discontinue capturingpackets after capturing a threshold number of packets, at the end of athreshold window of elapsed time, and/or at when triggered by a rule. Insome embodiments, offload/capture (e.g., by adding fields in the packetheader) CLDs 102A may capture all ingress traffic into a circularbuffer, and rules may be used to flag captured packets for laterretrieval and analysis. In certain embodiments, a triggering event maycause the state machine to walk back through the capture buffer toretrieve a specified number or time window of captured packets to allowlater analysis of the events leading up to the triggering event.

In certain embodiments, offload/capture CLDs 102A may keep a record oftriggering events external to the packet capture data for later use innavigating the packet capture data. This external data may operate as anindex into the packet capture data (e.g., with pointers into that data).

Efficient Packing of Packets in Circular Capture Memory

Existing packet capture devices typically set aside a fixed number ofbytes for each packet (16 KB for example). This is very inefficient ifthe majority of the packets are 64B since most of the memory is leftunfilled. The present disclosure is of a more efficient design in whicheach packet is stored in specific form of a linked list. Each packetwill only use the amount of memory required, and the link will point tothe next memory address whereby memory is packed with network data andno memory wasted. This allows the storage of more packets with the sameamount of memory.

In some embodiments, capture/offload CLDs 102A implement a circularcapture buffer capable of capturing ingress/egress packets storing eachin memory 103A. Some embodiments are capable of capturing ingress and/oregress packets at line rate. In some embodiments, memory 103A issubdivided into individual banks and each bank is assigned to anexternal network interface 101. In certain embodiments, networkinterface ports 101 are configured to operate at 10 Gbps and each portis assigned to a DDR2 memory interface. In certain embodiments, networkinterface ports 101 are configured to operate at 1 Gbps and two portsare assigned to each DDR2 memory interface. In these embodiments, thememory may be subdivided into two ranges exclusive to each port.

FIG. 26 illustrates the efficient packet capture memory system 600,according to certain embodiments of the present disclosure. This systemincludes functionality implemented in offload/capture CLD 102A workingin conjunction with capture buffer memory 103A. Offload/capture CLD 102Amay include capture logic 520 including decisional logic 604A and 604B,first in first out (FIFO) memories 606A and 606B, buffer logic 610, andtail pointer 612. Memory 103A may be a DDR2 or DDR3 memory module withaddressable units 608. Data in memory 103A may include a linked list ofrecords including Packets 1 through 4. Each packet spans a number ofaddressable units 608 and each includes a pointer 614 to the previouspacket in the list.

Circular Buffer/Packet Description

Below is a more detailed description of the data format of packet datain memory 103A, according to certain embodiments of the presentdisclosure. Packet data is written to memory 102A with a prepend header.The data layout for the first 32 Bytes of a packet captured in memory103A may contain 16 bytes of prepended header information and 16 bytesof packet data. Subsequent 32 byte blocks are written in a continuousmanner (wrapping to address 0x0 if necessary) until the entire packethas been captured. The last 32 byte block may be padded if the packetlength (minus 16 bytes in the first block) is not an integer multiple of32 bytes:

BOTH Egress/Ingress Packets [255:0]: Data[255:128] = first 16 bytes oforiginal packet data Data[127:93] = Reserved Data[91:64] = 28 bit DDR2address of previous packet for this thread (ingress/egress) Data[63:57]= DEFINED BELOW (Ingress/Egress definition) Data[56:43] Byte count (doesnot include 4 bytes of corrupted CRC if indicated) Data[42] = Threadtype (1 = ingress, 0 = egress) Data[41:40] = port number Data[39:0] = 40bit timestamp (10 ns resolution) Egress ONLY: Data [92] = ReservedData[63] = Corrupted CRC included in packet data (packet 4 bytes longerthan byte count) Data[62] = Corrupted IP checksum Data[61] = Packetrandomly corrupted Data[60] = Packet corrupted from byte 256 until theend of packet Data[59] = Packet corrupted in 65-255 byte range Data[58]= Packet corrupted in lower 64 bytes Data[57] = Packet fragmentedIngress ONLY: Data[92] = Previous packet caused circular buffer triggerData[63:62] = Reserved Data[61] = IP checksum good Data[60] = UDP/TCPchecksum good Data[59] = IP packet Data[58] = UDP packet Data[57] = TCPpacket

In certain embodiments, the following algorithm describes the process ofcapturing packet data. As a packet arrives at offload/capture CLD 102Avia internal interface 602, decisional logic 604A determines whether ornot to capture the packet in memory 103A. This decision may be based ona number of factors, as discussed elsewhere in this disclosure. Forexample, packet capture could be manually enabled for a specific windowof time or could be triggered by the occurrence of an event (e.g., apacket with an erroneous checksum value). In some embodiments, packetcapture is enabled by setting a specific bit in the memory of CLD 102A.If the packet is to be captured, the packet is stored locally in egressFIFO 606A. A similar process applies to packets arriving at externalinterface 101, though decisional logic 604B will store captured ingresspackets in ingress FIFO 606B. In each case, other processing may occurafter the packet arrives and before the packet is copied into the FIFOmemory. Specifically, information may be added to the packet (e.g., in aprepend header) such as an arrival timestamp and flags indicating thevalidity of one or more checksum values.

Buffer logic 610 moves packets from FIFOs 606A and 606B to memory 103A.Buffer logic 610 prioritizes the deepest FIFO to avoid a FIFO overflow.To illustrate the operation of buffer logic 610, consider the operationwhen packet capture is first enabled. In this initial state, both FIFOsare empty, tail pointer 612 is set to address 0x0, and memory 103A hasuniform value of 0x0. In embodiments where memory 103A may have aninitial value other than zero, capture/offload CLD 102A may storeadditional information indicating an empty circular buffer. Assume thatpacket capture is enabled.

At this time, an ingress packet arrives at external interface 101 and isassociated with an arrival timestamp and flags indicating checksumsuccess. Ingress decisional logic 604B creates the packet captureprepend header (the first 16 bytes of data described above) copies thepacket with its prepend header into FIFO 606B. Next, buffer logic 610copies the packet to the location 0x0, as this is the first packetstored in the buffer. In certain embodiments, memory 103A is DDR2 RAM,which has an effective minimum transfer unit of 256 bits, or 32 Bytes.In these embodiments, the packet is copied in 32 Byte units and the lastunit may be padded.

When another ingress packet arrives at external interface 101, ingressdecisional logic 604B follows the same steps and copies the packet withits prepend header into FIFO 606B. Next, buffer logic 610 determinesthat tail pointer 612 points to a valid packet record. The value of tailpointer 612 is copied into the prepend header of the current packet(e.g., at Data[91:64]) and tail pointer 612 is set to the address of thefirst empty block of memory 102B and buffer logic 610 copies the currentpacket to memory 103A starting at the address specified by tail pointer612.

In certain embodiments, ingress packets are linked separately fromegress packets as separate “threads” in the circular buffer. In theseembodiments, at least one additional pointer will be maintained in CLD102A in addition to tail pointer 610 to allow buffer logic 610 tomaintain linkage for both threads. In particular, if the buffer is notempty, tail pointer 612 points to a packet of a particular thread type(e.g., ingress or egress). If a new packet to be stored of the samethread type, the tail pointer may be used to set the previous packetpointer in the new packet to be stored. If the new packet to be storedis of a different thread type, buffer logic 610 will reference a storedpointer to the last packet of the different thread type to set theprevious packet pointer value on the new packet to be stored, but willstill store the new packet after the packet identified by tail pointer612.

Trigger Programming

In some embodiments, capture/offload CLD 102A may have three logiclayers of trigger programming. The first layer may allow up to fivecombinatorial inverted or non-inverted inputs of any combination of VLANID, source/destination IP address, and source/destination port addressto a single logic gate. All bits may be maskable in each of the fivefields to allow triggering on address ranges.

The first level may have four logic gates. Each of the four logic gatesmay be individually programmed to be a OR, NOR, or AND gate. The IPaddresses may be programmed to trigger on either IPV4 or IPV6 packets.The second level may have two gates and allow the combination ofnon-inverted inputs from the four first layer gates. These two secondlevel gates may be individually programmed for an OR, NOR, or AND gate.The third level logic may be a single gate that allows the combinationof non-inverted inputs from the four first layer gates and the twosecond level gates. This third level may be programmed for OR, NOR, orAND gate logic.

The logic may also allow for triggering on frame check sequence (FCS)errors, IP checksum errors, and UDP/TCP checksum errors.

Buffer Rewind

In some embodiments, CLD 102A may include rewind logic (e.g., as part ofbuffer logic 610) to generate a forward linked list in the process ofgenerating a properly formatted PCAP file. This rewind logic ispreferably implemented in CLD 102A due to its direct connection tomemory 103A. The rewind logic, when triggered, may perform an algorithmsuch as the following, written in pseudo code:

wrap = FALSE; // note if rewind wraps around the end of the memory next= tail; cur = tail.prev; prev = cur.prev; end_of_buffer = tail +packet_length(tail); while ( XOR (cur.prev < end_of_buffer, wrap) ) //invert test if buffer has wrapped     cur.prev = next; // reversepointer to next rather than     previous element in list     // shiftpointers to next element in list     next = cur;     cur = prev;    prev = cur.prev;     if (cur < prev) then wrap = TRUE; // test for awrap around     in memory end while

The rewind logic walks backward through the list starting at the tail,and changes each packet's previous pointer to be a next pointer, thuscreating a forward linked list. Once completed, the variable cur pointsto the head of a forward-linked list that may be copied to drive 109 forpersistent storage. Because the address 0x0 is a valid address, there isno value in checking for NULL pointers. Instead, buffer logic 610 shouldbe careful to not copy any entries after the last entry, identified bytail pointer 612.

Data Loopback and Capture

FIG. 27 illustrates two methods for capturing network data. Arrangement630 illustrates an in-line capture device with a debug interface. Thisarrangement is also called a “bump in the line” and can be inserted in amatter transparent to the other devices in the network. Arrangement 632is a network switch configured to transmit copies of packets transmittedor inject previously captured packets.

In some embodiments, network testing system 16 may provide data loopbackfunctionality, e.g., to isolate connectivity issues when configuringtest environments. FIG. 28 illustrates two loopback scenarios. Scenario634 provides a general illustration of an internal loopback implementedwithin a networking device that retains all networking traffic internalto that device. In conventional systems, loopback may be provided byconnecting a physical networking cable between two ports of the samedevice, in order to route data exiting the device back into the device,rather than sending the data to an external network or device. In such aconfiguration, all data sent by one port of the device is immediately(subject to speed of light delay) delivered to the other port and backinto the device. In system 16, internal loopback functionality may beprovided by a virtual wire loopback technique, in which data originatingfrom system 16 is looped back into the system 16 (without exiting system16), without the need for physical cabling between ports. Such techniqueis referred to herein as “virtual wire loopback.”

Scenario 636 provides a general illustration of an external loopbackimplemented outside a device, e.g., to isolate that device from networktraffic. In this arrangement, data from an external source is loopedback toward the external source or another external target, withoutentering the device. In some embodiments, system 16 may implement suchexternal loopback functionality in addition to virtual wire internalloopback and/or physical wire internal loopback functionality discussedabove.

In particular embodiments, system 16 provides internal loopback (virtualwire and/or physical wire loopback) and external loopback functionality,in combination with packet capture functionality, in a flexibleconfiguration manner to enable analysis of internal or external trafficfor comparison, analysis and troubleshooting (e.g., for latencyanalysis, timestamp zeroing, etc.).

FIG. 29 illustrates two general arrangements for data loopback andpacket capture in a capture buffer, according to certain embodiments ofsystem 16. Arrangement 640 illustrates an internal loopback with acapture buffer enabled. In this arrangement, the user can execute asimulated test scenario, export the capture buffer, and examine andvalidate the correctness of the traffic. This can be done without manualconfiguration of cables to save time and to avoid a physical presence atlocation of the network equipment. The user can also baseline the timingand latency of the traffic. With internal loopback enabled the returnpath is located before the physical layer transceiver modules soexternal latency information can be obtained by comparing to aconfiguration with a cabled loopback on the transceivers.

Arrangement 642 illustrates an external loopback with capture bufferenabled. In this arrangement, the network testing system becomes atransparent packet sniffer. All traffic can be captured as shown FIG.27, the in-line capture device. Diagnostic pings or traffic can be sentfrom the external network equipment to validate the network testingsystem. Network traffic may be captured and analyzed prior to an actualtest run before the network testing system is placed in-line. Byproviding the capability to move the capture interface point to bothinternal and external loopback paths and capture traffic of bothconfigurations in the same manner, system configuration and debug aresimplified.

In some embodiments, network testing system 16 may include a loopbackand capture system 650 configured to provide virtual wire internalloopback (and may also allow physical wire internal loopback) andexternal loopback, in combination with data capture functionality. FIG.30 illustrates aspects an example loopback and capture system 650relevant to one of the network processors 105 in system 16, according onone embodiment. Components of the example embodiment shown in FIG. 30correspond to the example embodiments of system 16 shown in FIGS. 14Aand 14B. FIG. 31 illustrates example data packet routing and/or capturefor virtual wire internal loopback and external loopback scenariosprovided by loopback and capture system 650, as discussed below.

As shown in FIGS. 30 and 31, system 650 may include a capture/offloadFPGA 102 a coupled to a pair of test interfaces 101A and 101B, a capturebuffer 103A, a network processor 105 via a routing FPGA 102 b, and atraffic generation FPGA 102 c. Control processor 106 is coupled tonetwork processor 105 and has access to disk drive 109. A loopbackmanagement module 652 having software or other logic for providingcertain functionality of system 650 may be stored in disk drive 109, andloopback logic 654 and capture logic 520 configured to implementinstructions from loopback management module 652, may be provided inFPGA 102 a.

Loopback management module 652 may be configured to send control signalsto capture/offload FPGA 102 a to control loopback logic 654 toenable/disable an internal loopback mode and to enable/disable anexternal loopback mode, and to capture logic 520 to enable/disable datacapture in buffer 103A. Such instructions from loopback managementmodule 652 may be generated automatically (e.g., by control processor106) and/or manually from a user (e.g., via a user interface of system16). Thus, a user (e.g., a developer) may control system 650 to placesystem 16 (or at least a relevant card 54) in an internal loopback mode,an external loopback mode, or a “normal” mode (i.e., no loopback), asdesired for various purposes, e.g., to execute a simulated testscenario, analyze system latency, calibrate a timestamp function, etc.

Thus, loopback logic 654 may be configured to control the routing ofdata entering capture/offload FPGA 102 a to enable/disable the desiredloopback arrangement. For example, with virtual wire internal loopbackmode enabled, loopback logic 654 may receive outbound data from networkprocessor 105 and reroute such data back to network processor 105 (or toother internal components of system 16), while capture logic 520 maystore a copy of the data in capture buffer 103A if data capture isenabled. The data routing for such virtual wire internal loopback isindicated in the upper portion of FIG. 31. As another example, loopbacklogic 654 may enable virtual wire internal loopback mode to provideloopback of data generated by traffic generation FPGA 102 c. Forinstance, loopback logic 654 may be configured in an internal loopbackmode to route data from traffic generation FPGA 102 c to networkprocessor 105 (or to other internal components of system 16), whilecapture logic 520 may store a copy of the data in capture buffer 103A ifdata capture is enabled, instead of routing data from traffic generationFPGA 102 c out of system 16 through port(s) 101. Control processor 106(and/or other components of system 16) may subsequently access captureddata from buffer 103A, e.g., via the Ethernet management networkembodied in switch 110 of system 16, for analysis.

In some embodiments, loopback logic 654 may simulate a physical wireinternal loopback, at least from the perspective of network processor105, for a virtual wire internal loopback scenario. FIG. 30 indicates(using a dashed line) the connection of a physical cable between testinterfaces 101A and 101B that may be simulated by such virtual wireinternal loopback scenario. For example, loopback logic 654 may adjustheader information of the looped-back data such that the data appears tonetwork processor 105 to have arrived over a different test port 101than the test port 101 that the data was sent out on. For example, ifnetwork processor 105 sends out data packets on port 0, loopback logic654 may adjust header information of the packets such that it appears tonetwork processor 105 that the packets arrived on port 1, as wouldresult in a physical wire loopback arrangement in which a physical wirewas connected between port 0 and port 1. Loopback logic 654 may providesuch functionality in any suitable manner. In one embodiment, loopbacklogic 654 includes a port lookup table 656 that specifies for eachegress port 101 a corresponding ingress port 101 for which networkprocessor 105 may expect data to be looped-back through in an internalloopback mode. For example, in a four port system, port lookup table 656may specify:

egress port ingress port 1 0 0 1 2 3 3 2

To implement port lookup table 656, with reference to FIG. 31, loopbacklogic 654 reads the egress port number (in this example, port 0) fromthe prepend header PH on each data packet P1 received from networkprocessor 105, determines the corresponding ingress port number (port 1)from table 656, and for each packet P1 inserts a new prepend header PH′that includes the determined ingress port number (port 1). Thus, whenpackets P1 are received at network processor 105, they appear to havereturned on port 1 (while in reality they do not even reach the ports).

Internal loopback mode (virtual or physical cable based) may be used forvarious purposes. For example, latency associated with system 16 and/oran external system (e.g., test system 18) may be analyzed by sending andreceiving data using system 16 with internal loopback mode disabled andmeasuring the associated latency, sending and receiving data usingsystem 16 with internal loopback mode enabled and measuring theassociated latency, and comparing the two measured latencies todetermine the extent of the overall latency that is internal to system16 versus external to system 16. As another example, internal loopbackmode (virtual or physical cable based) may be used to calibrate atimestamp feature of system 16, e.g., to account for inherent internallatency of system 16. In one embodiment, system 16 uses a 10 nanosecondtimestamp, and system 650 may use internal loopback to calibrate, or“zero,” the timestamp timing to 1/10 of a nanosecond. The zeroingprocess may be used to measure the internal latency and calibrate theprocess such that the timestamp measures the actual external arrivaltime rather than the time the packet propagates through to the timestamplogic. This may be implemented, for example, by enabling the internalloopback mode and packet capture. When an egress packet arrives atcapture/offload CLD 102A, the packet is time stamped and captured intopacket capture buffer 350. The egress packet is then converted by theinternal loopback logic into an ingress packet and time stamped on“arrival.” The time-stamped ingress packet is also stored in packetcapture buffer 350. The difference in time stamps between the egress andingress packet is the measure of internal round-trip latency. Thisability to measure internal latency can be especially valuable forconfigurable logic devices, where an image change may alter the internallatency.

As discussed above, loopback and capture system 650 may also provideexternal loopback functionality. That is, loopback management module 652may instruct loopback logic 654 to route data received on one port(e.g., port 0) back out over another port (e.g., port 1) instead offorwarding such data into system 16 (e.g., to network processor 105,etc.), as indicated in FIG. 31 with respect to packets P2. Also, as withinternal loopback mode, in external loopback mode, loopback managementmodule 652 may also instruct capture logic 520 to store a copy of datapassing through capture/offload FPGA 102 a in capture buffer 103A, alsoindicated in FIG. 31. Control processor 106 (and/or other components ofsystem 16) may subsequently access captured data from buffer 103A, e.g.,via the Ethernet management network embodied in switch 110 of system 16,for analysis of such captured data. Thus, using external loopback mode,system 16 may essentially act as a “bump in the wire” sniffer forcapturing data into a capture buffer.

Multi-Key Hash Tables

Standard implementations of hash tables map a single key domain to avalue or set of values, depending on how collisions are treated. Certainapplications benefit from a hash table implementation with multipleco-existent key domains. For example, when tracking network devicestatistics some statistics may be collected with visibility only intothe IP address of a device while others may be collected with visibilityonly into the Ethernet address of that device. Another exampleapplication is a host identification table that allows location of ahost device record by IP address, Ethernet address, or an internalidentification number. A hash table with N key domains is mathematicallydescribed as follows:

f₁:  K₁ → V f₂:  K₂ → V … f_(n):  K_(n) → V

An additional requirement is needed to ensure the above model representsa single hash table with N key domains instead of simply N hash tablesthat use the same value range:

-   -   If an entry y has a key k_(i) in domain K_(i), then all domains        K₁ through K_(n) must have a key k_(i) such that f_(j)(k_(j))        maps to the same entry y.

Standard hash table implementations organize data internally so that anentry can only be accessed with a single key. Various approaches existto extend the standard implementation to support multiple key domains.One approach uses indirection and stores a reference to the value in thehash table instead of the actual value. The model becomes this:

f₁:  K₁ → R f₂:  K₂ → R … f_(n):  K_(n) → R

In this model R is the set of indirect references to values in V, and alookup operation returns an indirect reference to the actual value,which is stored externally to the hash table. This approach has anegative impact on performance and usability. Performance degradationresults from the extra memory load and store operations required toaccess the entry through the indirect reference. Usability becomes achallenge in multithreaded environments because it is difficult toefficiently safeguard the hash table from concurrent access due to theindirect references.

Certain embodiments of the present disclosure support multipleindependent key domains, avoid indirect references, and avoid thenegative performance and usability impact associated with other designsthat support multiple key domains. According to certain embodiments ofthe present invention, each hash table entry contains a preciselyarranged set of links. Each link in the set is a link for a specific keydomain. In some embodiments, a software macro is used to calculate thedistance from each of the N links to the beginning of the containingentry. This allows the table to find the original object, much like amemory allocator finds the pointer to the head of a memory chunk.Defining the hash table automatically generates accessors to get theentry from any of N links inside the entry.

FIG. 32 illustrates a multiple domain hash table according to certainembodiments of the present disclosure. Hash table 680 includes bucketarray 682 with entries 684 pointing to linked list elements 686, 688,and 690. Linked list elements, e.g., 686, include pointers List1 andList2, and data including Key1 and Key2. List1 is associated with Key1and List2 is associated with Key2. Because entries 684 point to linkedlists, hash value collisions are handled by adding additional linkedlist elements to the list originating at the bucket array correspondingto the hash value.

Bucket array 682 may be an array of pointers, e.g., 32 bit or 64 bitaddresses, of length hash_length. Bucket array entries 684 a through 684c are identified as non-NULL entries, meaning that each contains a validpointer to a linked list element in memory. Bucket array entry 684 acontains a pointer to linked list element 686. Linked list element 686contains a pointer, e.g., the List1, to the next element in the linkedlist, if any. In FIG. 32, the List1 pointer in element 686 points to688. The List1 pointer in element 688 is NULL, indicating the end of thelist linked to bucket array entry 684 a.

FIG. 33 illustrates an example process 690 for looking up linked listelement 686 based on its Key1 value, according to certain embodiments ofthe present disclosure. Input Key1 of 686 into a hashing function toobtain index V₁. Index V₁ into bucket array 682 is bucket array entry684 a. Because that array entry is not NULL, follow the pointer andcheck each element in the linked list to see if the Key1 field of thatelement matches the Key1 value input into the hash value at the start ofthis process. Linked list element 686 is a match. Had Key1 of element686 not matched, the algorithm would follow the List1 pointer to linkedlist element 688 and would continue walking the linked list until itfound a match or a NULL pointer signaling the absence of a matchingentry. The prior art describes this approach for a single key value.

Similar to bucket array entry 684 a, entry 684 b points to linked listelement 686. However, in some embodiments of the present disclosure,entry 684 b points to the List2 pointer in linked list element 686,which then points to element 690.

FIG. 34 illustrates an example process 692 for looking up linked listelement 686 based on its Key2 value, according to certain embodiments ofthe present disclosure. Input Key2 of 686 into a hashing function toobtain index V₂. Index V₂ into bucket array 682 is bucket array entry684 b. Because that array entry is not NULL, follow the pointer andcheck each element in the linked list to see if the Key2 field of thatelement matches the Key2 value input into the hash value at the start ofthis process. Linked list element 686 is a match. However, to retrievethe element, the address in bucket 684 b should be adjusted upward thememory size of a pointer because bucket 684 b points to the secondrecord in that element. In some embodiments, three or more list pointersand associated key values are provided for.

Accordingly, linked list element 686 may be located in the same hashtable using two different key values without the use of indirection andwithout addition any additional storage overhead. Adding another key tothe same hash table merely requires the addition of two field entries inthe linked list element data structure: the list pointer and key value.

In some embodiments, this multikey hash table implementation relies ontwo or more sets of accessor functions. Each set of accessor functionsincludes at least an insert function and a lookup function. The lookupfunction for Key1 operates as illustrated in FIG. 33 and the lookupfunction for Key 2 operates as illustrated in FIG. 34. The insertfunctions operate in a similar fashion. The insert function for Key1performs the hash on Key1 of a new element and, if empty, points thebucket entry to the new element, or adds the new element to the end ofthe linked list. In some embodiments, the new element is added to thebeginning of the linked list. The insert function for Key2 performs thehash on Key2 of a new element and, if empty, points the bucket entry tothe List2 pointer of the new element, or adds the new element to linkedlist. The insert function for Key2 always points other entries to theList2 field of the new element rather than the start of that element.

In some embodiments, all sets of accessor functions use the same hashfunction. In other embodiments, one set of accessor functions uses adifferent hash function than a second set of accessor functions.

In certain embodiments, the accessor functions are generatedprogrammatically using C/C++ style macros. The macros automaticallyhandle the pointer manipulation needed to implement the pointer offsetsneeded for the second, third, and additional keys. A programmer needonly reference the provided macros to add a new key to the hash table.

Packet Assembly and Segmentation

Segmentation

The transmission control protocol (TCP) is a standard internet protocol(e.g., first specified in request for comments (RFC) 675 published bythe Internet Engineering Task Force in 1974). TCP generally aligns withLayer 4 of the Open Systems Interconnection (OSI) model of networkabstraction layers and provides a reliable, stateful connection fornetworking applications. As an abstraction layer, TCP allowsapplications to create and send datagrams that are larger than themaximum transmission unit (MTU) of the network route between the endpoints of the TCP connection. Networking systems support TCP bytransparently (to the application) segmenting over-sized datagrams atthe sending network device and reassembling the segments at thereceiving device. When a TCP channel is requested by an application, asetup protocol is performed wherein messages are sent between the twoend-point systems. Intermediate network nodes provide information duringthis process about the MTU for each link of the initial route that willbe used. The smallest reported MTU is often selected to minimizeintermediate segmentation.

Many network interface controllers (NICs) provide automatic segmentationof a large packet into smaller packets using specialized hardware priorto transmission of that data via an external network connection such asan Ethernet connection. The architecture of system 16 differs fromtypical network devices because network processors 105 share networkinterfaces 101 and are not directly assigned NICs with specializedsegmentation offload hardware. Further, network processors 105 do notinclude built-in TCP segmentation offload hardware. In order toefficiently handle TCP traffic, the present disclosure provides aCLD-based solution that post-processes jumbo-packets generated by thenetwork processor and splits those packets into multiple smaller packetsas specified in the header of a packet.

In certain embodiments of the present disclosure, the network processorincludes a prepend header to every egress packet. That prepend headerpasses processing information to offload/capture CLD 102A. Two fields inthe prepend header provide instructions for TCP segmentation. The firstis a 14 bit field that passes the packet length information in bytes(TCPsegLen). TCP lengths can in theory then be any length from a singlebyte to a max of 16 KB. The second field is a single bit that enablesTCP segmentation (TCPsegEn) for a given a packet.

FIG. 35 illustrates segmentation offload 700, according to certainembodiments of the present disclosure. Network processor 105 sendspacket 702 to capture/offload CLD 102A (e.g., via routing CLD 102B).Packet 702 includes a prepend header and a datagram. Segmentation logic704 includes logic to examine the prepend header and to segment thepacket into a series of smaller packets, which may be stored in outboundFIFO 708 for subsequent transmission via external interface 101.

FIG. 36 illustrates segmentation offload process 720, according tocertain embodiments of the present disclosure. When a start of packet(SOP) is received at step 722 from a network processor, segmentationlogic 704 is triggered. At step 724, segmentation logic 704 examines thepacket's prepend header to see if a segmentation flag (e.g., theTCPsegEn bit) is set. If not, the packet is passed along as is at step726.

If the segmentation flag is set, segmentation logic 704 determines thesegment length (e.g, by extracting the 14 bit TCPseglen field from theprepend header) and extracts the packet's IP and TCP headers at step728. Segmentation logic 704 may also determine whether the packet is anIPv4 or IPv6 packet and may verify that the packet is a properly formedTCP packet.

At step 730, segmentation logic 704 generates a new packet 706 the sizeof the segment length and copies in the original packet's IP and TCPheaders. Segmentation logic 704 may keep a segment counter and set asegment sequence number on new packet. Segmentation logic 704 may thenfill the data payload of new packet 706 with data from the data payloadportion of original packet 722. Segmentation logic 704 may update the IPand TCP length fields to reflect the segmented packet length andgenerate IP and TCP checksums.

Once new packet 706 has been generated, that packet may be added to afirst in first out (FIFO) queue at step 732 for subsequent transmissionvia external interface 101. At step 734, segmentation logic 704 maydetermine whether any new packets are needed to transmit all of the datafrom original packet 702. If not, the process stops at step 736. If so,step 730 is repeated. At step 730, if less data remains than can fillthe data portion of a packet of length segment length, a small packetmay be generated rather than padding the remainder.

Assembly

When TCP packets arrive at system 16, they may arrive as segments of anoriginal, larger TCP packet. These segments may arrive in sequence ormay arrive out of order. While a CPU is capable of reassembling thesegments, this activity consumes expensive interrupt processing time.Many operating systems are alerted to the arrival of a new packet whenthe NIC triggers an interrupt on the CPU. Interrupt handlers often runin a special protected mode on the processor and switching in and out ofthis protected mode may require expensive context switching processes.Conventional systems offload TCP segment reassembly to the networkinterface card (NIC). However, these solutions require shared memoryaccess between the receiving processor and its network interface card(NIC). Specifically, some commercially-available NICs manipulate packetbuffers in shared memory mapped between the host CPU and the NIC. System16 has no memory shared memory between the network processor and routingCLD 102B. Furthermore, a conventional PCI bus and memory architecturedoes not provide sufficient bandwidth to enable reassembly at the linerates supported by system 16. In the present disclosure, a TCPreassembly engine is provided in a CLD between external interfaces 101and the destination network processor. This reassembly engine forwardsTCP segment “jumbograms” to the network processor rather than individualsegmented packets. The operation of the reassembly engine can reduce thenumber of packets processed by the NP by a factor of 5, which frees upsignificant processing time for performing other tasks.

FIG. 37 illustrates packet assembly system 740. The packet assemblysystem includes routing CLD 102B and memory 103A. Routing CLD 102Bincludes assembly logic 744, which processes packet 742 received fromexternal interface 101 (e.g., via offload/capture CLD 102A). Memory 103Aincludes packet record array 746, which contains pointers to linkedlists of packet segments 748. In some embodiments, packet record array746 may be in internal memory within CLD 102B. In some embodiments,packet assembly logic 744 may selectively forward received packet 742 tonetwork processor 105 as-is, as a set of a partially reassembled TCPjumbogram, or as a fully reassembled jumbogram. In certain embodiments,received packets are queued in receive FIFO 750 and packets forwarded tonetwork processor 105 are queue in transmit FIFO 752.

Network processor 105 may control the operation of assembly logic 744 byaltering configuration parameters on the reassembly process. In someembodiments, network processor 105 may control the number of receivebucket partitions in memory 103B and/or the depth of each receive bucketpartition. In certain embodiments, network processor 105 may selectivelyroute certain packet flows through or around the assembly engine basedon at least one of the subnet, VLAN, or port range.

FIG. 38 illustrates process 760 performed by receive state machine (Rx)in assembly logic 744, according to certain embodiments of the presentdisclosure. The receive state machine monitors receive FIFO at step 762.When a packet arrives (at step 764), the packet is examined to determinewhether it is a segment of a TCP jumbogram. If not, the packet is queuedin transmit FIFO (at step 766) for delivery to network processor 105. Ifthe packet is a segment, the receive state machine may apply a bypassfilter (at step 768) to determine whether assembly should be attempted.If not, the packet is queued for transmission as-is. If assembly shouldbe attempted, the packet is compared to packet assembly records 746 (atstep 770) to identify a matching packet segment bucket. This comparisonprocess may include extraction of a 4-tuple of the IP source address, IPdestination address, IP source port, and IP receive port. This 4-tuplemay be sorted and input into a hash function (e.g., jhash) to generate ahash value. That hash value may be used to index into the packetassembly records array 746.

If a match is found, the packet is added to the matching bucket (at step772). Receive state machine may insert the new packet into linked list748 in the appropriate ordered location based on the packet's TCPsequence number. Receive state machine also checks whether this newestpacket completes the sequence for this TCP jumbogram (at step 774). Ifso, receive state machine sets the commit bit on the correspondingpacket assembly record 746 (at step 776). If the newest packet does notcomplete the sequence (at step 778), receive state machine updates thecorresponding packet assembly record 746 and stops.

If no matching packet assembly record 746 was found (at step 770), then,space permitting, receive state machine creates a new record (at step780) and adds the received packet to the newly assigned reassemblybucket list (at step 782).

FIG. 39 illustrates process 800 performed by transmit state machine (Tx)in assembly logic 744, according to certain embodiments of the presentdisclosure. Transmit state machine continually monitors each bucket inthe assembly memories 103B (at step 802). The transmit state machinechecks to see if the bucket is empty (at step 804). If the bucket is notempty, the following conditions are checked (at step 806) to determineif the packet should be committed to the network processor:

1. Commit bit is set. This bit can be set by the receive state machine.

2. Current time—Packet initial timestamp>Age-out value.

3. When only 1 free bucket remains, then the bucket with the oldesttimestamp will be committed.

When a packet is being committed, the transmit state machine will setthe lock bit on the packet assembly record marking it unavailable. Ifthe packet is complete (at step 808), the transmit state machine willassemble (at step 810) a TCP jumbogram including the IP and TCP headersof, for example, the first packet in the sequence (after stripping outthe TCP segmentation related fields), and the concatenated data portionsof each segment packet. The transmit state machine (at step 812) addsthe newly assembled TCP jumbogram to the transmit FIFO and clears thepacket assembly record from memory 103B making it available for use bythe receive state machine.

If the packet is not complete, but the current packet aged out or wasforced out as the oldest packet in memory 103B, then transmit statemachine (at step 814) may move each packet segment as-is to transmitFIFO 752 and clear out the corresponding packet assembly record.

32-Bit Pointer Implementation for 64-Bit Processors

On 64-bit systems pointers typically consume 8 bytes of computer memory(e.g., RAM). This is double the amount needed on 32-bit systems and canpose a challenge when migrating from a 32-bit system to a 64-bit system.

Typical solutions to this problem include: increasing the amount ofavailable memory, and rewriting the software application to reduce thenumber of pointers used in that application. The first solution listedabove is not always possible. For example, when shipping software-onlyupgrades to hardware systems already deployed at customer sites. Thesecond solutions can be cost prohibitive and may not reduce memoryrequirements enough to enable the use of 64-bit pointers.

The system of the present disclosure specially aligns the virtual memoryoffsets in the operating system so that virtual addresses all fall underthe 32 GB mark. This means that for pointers, the upper 29 bits arealways zero and only the lower 35 bits are needed to address the entirememory space. At the same time, the system aligns memory chunks to an8-byte alignment. This ensures that the lower 3 bits of an address arealso zero.

As a result of these two implementation details, it is possible totransform a 64-bit pointer to a 32-bit pointer by shifting right 3 bits,and discarding the upper 32-bits. To turn the compressed address back toa 64-bit real address, one simply shifts the 32-bit address left by 3bytes and stores in a 64-bit variable. Certain embodiments of thepresent disclosure may extend this approach to address 64 GB or 128 GBof memory by aligning memory chunks to 16 or 32-byte chunks,respectively.

Task Distribution

In some embodiments, system 16 comprises a task management engine 840configured to allocate resources to users and processes (e.g., tests ortasks) running on system 16, and in embodiments that include multiplenetwork processors 105, to distribute such processes among the multiplenetwork processors 105 to provide desired or maximized usage ofresources.

Definitions of certain concepts may be helpful for a discussion of taskmanagement engine 840. A “user” refers to a human user that has invokedor wishes to invoke a “test” using system 16. One or more users may runone or more tests serially or in parallel on one or more networkprocessors 105. A test may be defined as a collection of “tasks” (alsocalled “components”) to be performed by system 16, which tasks may bedefined by the user. Thus, a user may specify the two tasks “FTPsimulation” and “telnet simulation” that define an example test “FTP andtelnet simulation.” Some other example tasks may include SMTPsimulation, SIP simulation, Yahoo IM simulation, HTTP simulation, SSHsimulation, and Twitter simulation.

Each task (e.g., FTP simulation) may have a corresponding “taskconfiguration” that specifies one or more performance parameters forperforming the task. An example task configuration may specify twoperformance parameters: 50,000 sessions/second and 100,000 simulations.The task configuration for each task may be specified by the requestinguser, e.g., by selecting values (e.g., 50,000 and 10,000) for one ormore predefined parameters (e.g., sessions/second and number ofsimulations).

Some example performance parameters for a traffic simulation task areprovided below:

-   -   Data Rate Unlimited: defines whether data rate limiting should        be enabled or disabled for the test. Choose this option for        maximum performance or when a test's data rate is naturally        limited by other factors such as session rate. This option can        be useful for determining the natural upper-bound for a        performance test.    -   Data Rate Scope: defines whether the rate distribution number is        treated as a per-interface limit or an aggregate limit on the        traffic that this component generates. Because of the asymmetric        nature of most application protocols, when per-interface        limiting is enabled, client-side bandwidth is likely to be less        than server-side bandwidth. This means that the aggregate        bandwidth used for some protocols will be less than the sum of        the max allowed per interface. If you need a fixed amount of        throughput, use the aggregate limit.    -   Data Rate Unit: defines the units, either ‘Frames/Second’ or        ‘Megabits/Second’ that the Minimum/Maximum data rates (below)        represent.    -   Data Rate Type: ‘Constant’ indicates that all generated traffic        will be at the data rate specified by the Minimum data rate        field, ‘Range’ indicates that data rate should start at either        the Minimum or Maximum data rate and increase or decrease over        the course of the test, ‘Random’ indicates that data rate should        be chosen randomly between Minimum and Maximum data rates,        inclusive, changing once every tenth of a second during test        execution.    -   Minimum/Maximum data rate: min/max data rate. Values of 1 to        1488095 (1 Gigabit ports) or 14880952 (10 Gigabit ports) are        supported for ‘Frames/Second’. Values of 1 to 1000 (1 Gigabit        ports) or 10000 (10 Gigabit ports) are supported for        ‘Megabits/Second’.    -   Ramp Up Behavior:        -   During the ramp up phase, TCP sessions are only opened, but            no data is sent. This is useful for quickly setting up a            large number of sessions without wasting bandwidth. This            parameter defines what the test actually does during the            ramp up phase. Note: after the ramp up phase, all sessions            will fully open, even if the ramp up behavior was set to            something other than “Full Open”.        -   “Full Open”—The full TCP handshake is performed on open        -   “Full Open+Data”—Same as full, but start sending data        -   “Full Open+Data+Full Close”—Same as full+data, but also do a            full close for completed sessions.        -   “Full Open+Data+Close with Reset”—Same as full+data, but            also initiate the TCP close with a RST.        -   “Half Open”—Same as full, but omit the final ACK        -   “SYN Only”—Only SYN packets are sent        -   “Data Only”—Only PSH data packets are sent, with no TCP            state machine processing. This mode is not compatible with            SSL nor with Conditional Requests. Any flow using SSL will            send no packets.    -   SYN Only Retry Mode: defines the behavior of the TCP Retry        Mechanism when dealing with the initial SYN packet of a flow,        the following modes are permitted:        -   “Continous”—Continue sending SYN packets, even if we have            ran out of retries (Retry Count).        -   “Continous with new session”—Same as “Continous”, except we            change the initial sequence number every “Retry Count”            loop(s).        -   “Obey Retry”—Send no more than “Retry Count” initial SYN            packets.    -   Maximum Super Flows Per Second: defines the maximum number of        Super Flows that will be instantiated per second. If there is        one flow per Super Flow, as in Session Sender, this is        functionally equivalent to the sum of TCP and UDP flows per        second. In cases where there are multiple flows per Super Flow,        you may see a varying number of effective flows per second.    -   Maximum Simultaneous Super Flows: defines the maximum        simultaneous Super Flows that will exist concurrently during the        test duration. If there is one flow per Super Flow, as in        Session Sender, this is functionally equivalent to the sum of        TCP and UDP flows. In cases where there are multiple flows per        Super Flow, you may see a varying number of effective        simultaneous flows. This value defines a shared resource between        different test components, and is limited to 15,000,000. In        other words, the total maximum simultaneous sessions for all        components in a test will be less than or equal to 15,000,000.    -   Engine Selection: This parameter selects the type of engine with        which to run the test component. Select “Advanced” to enable the        default, full-featured engine. Select “Simple” to enable a        simpler, higher-performance, stateless engine.    -   Performance Emphasis: This parameter adjusts whether the        advanced engine's flow scheduler favors opening new sessions,        sending on existing sessions, or a mixture of both. Select        “Throughput” to emphasize sending data on existing sessions.        Select “Simultaneous Sessions” to emphazise opening new        sessions. Select “Balanced” to emphasize both equally—this is        the default setting.    -   Statistic Detail: This parameter adjusts the level of statistics        to be collected. Decreasing the number of statistics collected        can increase performance and allow for targeted reporting.        Select “Maximum” to enable all possible statistics. Select        “Application Only” to enable only Application statistics (L7).        Select “Transport Only” to enable only Transport statistics        (L4/L3). Select “Minimum” to disable most statistics    -   Unlimited Super Flow Open Rate: determines globally how fast        sessions are opened. If set to true, sessions will be opened as        fast as possible. This setting is useful for tests where the        session rate is not the limiting factor for a test's        performance. Note: this setting may produce session open rates        faster than the global limit.    -   Unlimited Super Flow Close Rate: determines how fast sessions        are closed. If set to false, session close rate will mirror the        session open rate. If set to true, sessions will be closed as        fast as possible.    -   Target Minimum Super Flows Per Second: specifies a minimum        number of sessions that the test must open in order to pass in        the final results. This is an aid for the user to define        pass/fail criteria for a particular test. This parameter does        not affect the network traffic of the test in any way.    -   Target Minimum Simultaneous Super Flows: specifies a minimum        number of sessions per second that the test must open in order        to pass in the final results. This is an aid for the user to        define pass/fail criteria for a particular test. This parameter        does not affect the network traffic of the test in any way.    -   Target Number of Successful matches: specifies the minimum        number of successful matches required to pass in the final        results. This is an aid for the user to define pass/fail        criteria for a particular test. This parameter does not affect        the network traffic of the test in any way.    -   Streams Per Super Flow: The maximum number of streams that will        be instantiated for an individual Super Flow at one time. The        effective number may be limited by the number of Super Flows in        the test. Setting this to a lower number makes tests initialize        faster and provides less-random application traffic. Setting        this to a higher number causes test initialization to take more        time, but with the benefit of more randomization, especially for        static flows.    -   Content Fidelity: Select “High” Fidelity to generate more        dynamic traffic. Select “Normal” Fidelity to generate simpler,        possibly more performant, traffic.

Each task requires a fixed amount of “resources” to complete the task. A“resource” refers to any limited abstract quantity associated with anetwork processor 105, which can be given or taken away. Exampleresources include CPU resources (e.g., cores), memory resources, andnetwork bandwidth. Each network processor 105 has a fixed set ofresources.

A “port” refers to a test interface 101 to the test system 18 beingtested. Thus, in example embodiments, a particular card 54 may have fouror eight ports (i.e., interface 101) that may be assigned to user(s) bytask management engine 840 for completing various tests.

In a conventional system, when test that requires certain resources isstarted, such resources may be available at the beginning of a test butthen become unavailable at some point during the test run, e.g., due toother tests (e.g., from other users) being initiated during the testrun. This may be particularly common during long running tests. Whenthis situation occurs, the test may have to be stopped or paused, as therequired resources for continuing the test are no longer available.Thus, it may be desirable to pre-allocate resources for each user sothat it can be determined before starting a particular test if theparticular test can run to completion without interruption. Thus, taskmanagement engine 840 may be programmed to allocate resources to usersand/or processes (tests and components thereof (i.e., tasks)) beforesuch processes are initiated, to avoid or reduce the likelihood of suchprocesses being interrupted or cancelled due to lack of resources.

Allocation of Resources to Users

In some embodiments, task management engine 840 is programmed toallocate resources to users based on a set of rules and algorithms(embodied in software and/or firmware accessible by task managementengine 840). In a particular embodiment, such rules and algorithmsspecify the following:

Rules:

1. Each user is allowed to reserve one or more ports 101 on a board 54.

2. Only one user may reserve any given port 101.

3. The resources on a particular board 54 allocated to each usercorrespond to the number of ports 101 on the board 54 allocatedto/reserved by that user.

4. If all ports 101 on a board 54 are allocated to/reserved by aparticular user, then all resources of that board 54 allocatedto/reserved by that user. For example, if a user reserves 2 of 8 portson a board, then 25% of all resources of that board are allocated tothat user.

In view of these rules, task management engine 840 is programmed withthe following algorithm for allocating the resources of a board 54 toone or more users.

Givens:

-   -   Let “U” denote the set of all users.    -   Let “NP” denote the set of all network processors 105 on the        board 54.    -   Let “n” denote the number of ports 101 controlled by the network        processors 105.    -   Let “K” denote the set of all possible abstract resources used        by all network processors 105.    -   Let “NPR(z,r)” denote the amount of resource “r” that a        particular network processor “z” currently has available, where        “r” is a member of set “K” and “z” is a member of set “NP.” The        amounts are in abstract units relevant to the particular network        processor.    -   Let “p(u)” denote the number of ports 101 reserved by a user        “u,” where u is a member of set “U.” Algorithm:

The algorithm getMaxResourceUtilization( ) computes the amount of eachresource “r” available to a given user. The total amount of any givenresource “r” will be the sum of that resource “r” across all networkprocessors 105 on the board 54. Thus, the algorithmgetMaxResourceUtilization( ) returns an array “UR(u,r)” where “r” is amember of “K” and “u” is a member of “U”. Each element of the arrayrepresents the amount of the resource available to the user. Thealgorithm is as follows:

begin getMaxResourceUtilization( )     set UR equal to { }     for eachr in K         # R is the total amount of resource “r” among         allnetwork processors.         set R = 0         for each z in NP            set R = R + NPR(z,r)         end for         # Distribute Ramong the users.         for each u in U             set UR(u,r) = R *p(u) / n         end for     end for     return UR end

FIG. 40 illustrates an example method 850 for allocating resources ofnetwork processors 105 in a system 16 to users, according to an exampleembodiment. At step 852, users submit requests to reserve testinterfaces (or “ports”) 101 for performing various tests of a testsystem 18. Users may submit such requests in any manner, e.g., via auser interface shown in FIG. 13A provided by system 16. Requests fromdifferent users may be made at different times. At step 854, taskmanagement engine 840 may assign ports 101 to users based on (a) portreservation requests made at step 852, (b) the number of currentlyavailable (i.e., unassigned) ports 101, and/or (c) one or more rules,e.g., a port reservation limit that applies to all users (e.g., eachuser can reserve a maximum of n ports at any given time), or portreservation limits based on the type or level of user (e.g., managerscan reserve a maximum of 8 ports at any given time, while technicianscan reserve a maximum of 4 ports at any given time).

At step 856, task management engine 840 may assign resources of networkprocessors 105 to users based on the number of ports 101 assigned toeach user by executing algorithm getMaxResourceUtilization( ) discussedabove. As discussed above, task management engine 840 may assign thetotal quantity of each type of network processor resource to users on apro rata basis, based on the number of ports assigned to each user. Forexample, if a user reserves 3 of 4 ports on a board, then 75% of eachtype of resource is assigned to that user.

Distribution of Tasks Across Network Processors

As discussed above, in some embodiments task management engine 840 isfurther programmed to distribute tasks (i.e., components of tests) amongthe multiple network processors 105 of system 16 to provide desired ormaximized usage of resources, and to determine whether a particular testproposed or requested by a particular user can be added to the currentlyrunning tests on system 16. In particular, task management engine 840may be programmed to distribute tasks based on a set of rules andalgorithms (embodied in software and/or firmware accessible by taskmanagement engine 840). For example, such rules and algorithms specifythe following:

Rules:

1. Each test is divided into tasks (also called components) that run inparallel.

2. Each task runs on a particular board 54 depending on the ports 101used by that task.

3. A task may not span more than one board 54.

4. Each task will consume a fixed quantity of resources “r.”

5. Each task on a board 54 will be assigned a particular networkprocessor 105 based on the resource usage of that task and the resourceson that board 54 allocated to the user.

In view of these rules, task management engine 840 is programmed withthe following algorithm for allocating the resources of a board 54 toone or more users.

Givens:

-   -   Let “T” denote the set of all current running tests.    -   Let “nt” denote the proposed test to add to set “T.”    -   Let “UT(t)” denote the user associated with test “t.”    -   Let “UC(t)” denote the set of tasks for test “t.”    -   Let “NPZ(t,c)” denote the network processor 105 associated with        task “c” of test “t.”    -   Let “X(t,c,r)” represent the amount of resource “r” used by task        “c” in test “t.”    -   Let “NPR(z,r)” denote the amount of resource “r” that a        particular network processor “z” currently has available, where        “r” is a member of set “K” and “z” is a member of set “NP.” The        amounts are in abstract units relevant to the particular network        processor.    -   Let “Q” denote the resources currently available to each user,        which may be defined for each user as the maximum resources        available to that user, i.e., UR(u,r) determined by the        algorithm getMaxResourceUtilization( ), minus any resources        currently used by that user.    -   Let “W” denote the resources currently available to each network        processor 105, which may be defined for each network processor        105 as the maximum resources available to that network processor        105, i.e., NPR(z,r) discussed above, minus any resources        currently used by that network processor 105.

The following table defines and provides examples for the variables usedin the task distribution algorithm addRunningTest( ):

Variable Definition Example T Set of all current running tests t1 t2 t3UT(t) 1D array indexed by test, contains index t1: u1 of user t2: u1 t3:u2 UC(t) 1D array indexed by test, contains set of t1: c1, c2, c3 tasks“c” of each test “t” in set T t2: c4 t3: c5, c6 NPZ(t, c) 2D arrayindexed by task index and test c1 c2 c3 c4 c5 c6 index, contains indexof the network t1: np1 np1 np1 np1 np1 np1 processor associated withtask “c” and t2: np1 np1 np1 np1 np2 np2 test “t” t3: np2 np2 np2 np2np2 np2 NNPZ(t, c) 2D array indexed by task index and test c1 c2 c3 c4c5 c6 index, contains index of the network t1: np1 np1 np1 np1 np1 np1processor associated with task “c” and t2: np1 np1 np1 np1 np2 np2 test“t,” including entries for newly t3: np2 np2 np2 np2 np2 np2 added test“nt” nt: np2 np2 np2 np2 np2 np2 X(t, c, r) 3D array indexed by resourceindex, task index, test index. Contains the t1: amount of resource “r”used by each c1 c2 c3 task “c” of each test “t” r1:  5% 10% 30% r2: 20%15% 25% r3: 15% 10% 20% t2: c4 r1: 15% r2: 30% r3: 20% t3: c5 c6 r1:  3% 6% r2:  5% 10% r3:  8%  5% NPR(z, r) 2D array indexed by resourceindex, and NP1 NP2 network processor index. Contains CPU: 30% 50% amountof resource “r” currently memory: 20% 60% available on each networkprocessor “z” bandwidth: 25% 70% K 1D array of all possible resources ofall Total CPU resources network processors “z” Total memory resourcesTotal n/w bandwidth resources r resource (member of set K) W resourcesavailable for each network processor z Q maximum resources available toeach user t test (member of set T) c task (component of a test “t”) ntnew test to be added to current set of running tests T NP set of allnetwork processors z network processor (member of set NP) u user

Algorithm:

The algorithm addRunningTest( ) determines whether to add (and if so,adds) a proposed test to a list of currently running tests on system 16.The algorithm addRunningTest( ) assumes that the resources used by therunning tests do not exceed the total resources available on system 16.The algorithm first determines all resources consumed by running testson system 16. The algorithm then determines whether it is possible toadd all of the tasks of the test to one or more network processors 105without exceeding (a) any quotas placed on the user (e.g., as specifiedfor the user in the user resource allocation array UR(u,r) determined asdescribed above), or (b) the maximum resources available to the relevantnetwork processor(s) 105.

If it is impossible to add any task of the proposed test to any networkprocessor based on the conditions discussed above, the algorithmdetermines not to add the test to the set of tasks running on system 16,and notifies the user that the test cannot be run. Otherwise, if alltasks of the proposed test can be added to system 16, the test is addedto the list of running tests, and the tasks are assigned to theirspecified network processor(s) 105, as determined by the algorithm.

The algorithm is as follows:

begin addRunningTest(nt)     # Determine the current resources availableto each user.     set Q = getMaxResourceUtilization( )     # Determinethe current resources available to each     network processor.     set W= NPR     # Subtract resources used by the current running tests     #from the resources available to the user and each     network processor.    for each test t in T         set u = UT(t)         for each c inUC(t)             set z = NPZ(t,c)             for each r in K                # subtract the amount from the                 # totalavailable to the user, and                 the total available to theprocessor.                 Q(u,r) = Q(u,r) − X(t)(c)(r)                W(z)(r) = W(z)(r) − X(t)(c)(r)             end for        end for     end for     # Assign new tasks to a networkprocessor     set u = UT(nt)     set NNPZ = { }     for each c in UC(nt)        # If the limit for any resource is exceeded, then fail        for each r in K             if X(nt,c,r) > Q(u,r)            then fail         end for         # Look for any networkprocessor that can         # accommodate the resource request        set found = false         for each z in NP             setall.ok = true             for r in K                 if X(nt,c,r) >W(z,r)                 then all.ok = false             end for            if all.ok then                 set found = true                set foundz = z             end if         end for        if not found, then fail         # Assign the task to a networkprocessor, and         # subtract its amount from the total available tothe         # user, and the total available to the processor.        set NNPZ(c) = foundz         for each r in K             Q(u,r)= Q(u,r) − X(nt)(c)(r)             W(foundz,r) = W(foundz,r) −X(nt)(c)(r)         end for     end for     # all tasks were assigned,we can add the test.     add t to T     for each c in UC(t)         setNPZ(t,c) = NNPZ(c)     end for end

FIGS. 41A-41E illustrate a process flow of the addRunningTest(nt)algorithm executed by task management engine 840, as disclosed above.

FIG. 41A illustrates a module 860 of the addRunningTest(nt) algorithmthat determines the current resources available to each user, Q, and thecurrent resources available to each network processor, W.

FIG. 41B illustrates a module 862 of the addRunningTest(nt) algorithmthat determines whether any of the tasks of the proposed new test wouldexceed the current resources available to the user that has proposed thenew test, as determined by algorithm module 860.

FIG. 41C illustrates a module 864 of the addRunningTest(nt) algorithmthat determines whether the network processors can accommodate the tasksof the proposed new test, based on a comparison of the current resourcesavailable to each network processor determined by algorithm module 860and the resources required for completing the proposed new task.

FIG. 41D illustrates a module 866 of the addRunningTest(nt) algorithmthat assigns the tasks of the proposed new task to one or more networkprocessors, if algorithm module 864 determines that the networkprocessors can accommodate all tasks of the proposed new test.

FIG. 41E illustrates a module 868 of the addRunningTest(nt) algorithmthat adds the proposed new test to the set of tests, T, running onsystem 16. Task management engine 840 may then instruct controlprocessor 106 and/or relevant network processors 105 to schedule andinitiate the new test.

FIG. 42 illustrates an example method 870 for determining whether a testproposed by a user can be added to the current list of tests running onsystem 16 (if any), and if so, adding the test to the list of currentlyrunning tests on system 16 and distributing the tasks of the proposedtest to one or more network processors 105 of system 16. At step 872, auser submits a request to run a new test on system 16, e.g., to testoperational aspects of a test system 18. The users may submit such newtest request in any manner, e.g., via a user interface shown in FIG. 13Dprovided by system 16.

In one embodiment, the user may define the proposed new test by (a)selecting one or more tasks to be included in the new test, e.g., byselecting from a predefined set of task types displayed by engine 840(e.g., FTP simulation, telnet simulation, SMTP simulation, SIPsimulation, Yahoo IM simulation, HTTP simulation, SSH simulation, andTwitter simulation, etc.), and (b) for each selected task, specifyingone or more performance parameters, e.g., by selecting any of theexample performance parameter categories listed above (Data RateUnlimited, Data Rate Scope, Data Rate Unit, Data Rate Type,Minimum/Maximum data rate, Ramp Up Behavior, etc.) and entering orselecting a setting or value for each selected performance parametercategory. Thus, for a telnet simulation tasks, the user may define theperformance parameters of 50,000 sessions/second and 100,000simulations.

At step 874, engine 840 may determine the amount of each type of networkprocessor resource “r” required for achieving the performance parametersdefined (in the relevant task configuration) for each task of theproposed new test, indicated as X(nt,c,r) in the algorithm above. Forexample, for a particular task of the new test, engine 840 may determinethat the task requires 20% of the total CPU resources of networkprocessors 105, 25% of the total memory resources of network processors105, and 5% of the total network bandwidth resources of networkprocessors 105. Engine 840 may determine the required amount of eachtype of network processor resource “r” in any suitable manner, e.g.,based on empirical test data defining correlations between particulartest performance parameters can empirically determined network processorresource quantities used by the relevant network processor(s) forachieving the particular performance parameters. In some instances,engine 840 may interpolate/extrapolate or otherwise analyze suchempirical test data to determine the network processor resourcesX(nt,c,r) required for achieving the performance parameters of theparticular task of the new test. In some embodiments, engine 840 maynotify the user of the required network processor resources determinedat step 874.

Task management engine 840 may then execute the addRunningTest(nt)algorithm disclosed above or other suitable algorithm to determinewhether the proposed new test can be added to the set of currentlyrunning tests on system 16 (i.e., whether all tasks of the proposed newtest can be added to system 16). At step 876, engine 840 may determinethe current resources available to each user (or at least the currentresources available to the requesting user) and the current resourcesavailable to each network processor, e.g., by executing algorithm module860 shown in FIG. 41A. In some embodiments, engine 840 may display orotherwise notify the user of the current resources available to thatuser, e.g., by displaying the current resources on a display.

At step 878, engine 840 may determine whether any of the tasks of theproposed new test would exceed the current resources available to therequesting user, e.g., by executing algorithm module 862 shown in FIG.41B. This may include a comparison of the required resources for eachtask as determined at step 874 with the current resources available tothe requesting user as determined at step 876. If any of the tasks ofthe proposed new test would exceed the requesting user's currentlyavailable resources, the proposed new test is not added to system 16, asindicated at step 880. In some embodiments, engine 840 may display orotherwise notify the user of the results of the determination. At step882, engine 840 may determine whether the network processors 105 canaccommodate the tasks of the proposed new test, e.g., by executingalgorithm module 864 shown in FIG. 41C. This may include a comparison ofthe required resources for each task as determined at step 874 with thecurrent resources available to each network processor as determined asdetermined at step 876. If it is determined that the network processors105 cannot accommodate the new test, the proposed new test is not addedto system 16, as indicated at step 880. In some embodiments, engine 840may display or otherwise notify the user of the results of thedetermination.

At step 884, if algorithm module 864 determines that the networkprocessors can accommodate all tasks of the proposed new test, engine840 assign the tasks of the proposed new task to one or more networkprocessors 105, e.g., by executing algorithm module 866 shown in FIG.41D. In some embodiments, engine 840 may display or otherwise notify theuser of the assignment of tasks to network processor(s). At step 886,engine 840 may adds the proposed new test to the set of tests running onsystem 16, e.g., by executing algorithm module 868 shown in FIG. 41E. Atstep 888, task management engine 840 may then instruct control processor106 and/or relevant network processors 105 to initiate the new test. Insome embodiments, engine 840 may notify the user of the test initiation.

Dynamic Latency Analysis

In some embodiments, network testing system 16 may perform statisticalanalysis of received network traffic in order to measure the quality ofservice provided under a given test scenario. One measure of quality ofservice is network performance measured in terms of bandwidth, or thetotal volume of data that can pass through the network, and latency(i.e., the delay involved in passing that data over the network). Eachdata packet passing through a network will experience its own specificlatency based on the amount of work involved in transmitting that packetand based on the timing of its transmission relative to other events inthe system. Because of the huge number of packets transmitted on atypical network, measurement of latency may be represented usingstatistical methods. Latency in network simulation may be expressed inabstract terms characterizing the minimum, maximum, and average measuredvalue. More granular statistical analysis may be difficult to obtain dueto the large number of data points involved and the rate at which newdata points are acquired.

In some embodiments, a network message may be comprised of multiplenetwork packets and measurement may focus on the complete assembledmessage as received. In some testing scenarios, the focus of theanalysis may be on individual network packets while other testingscenarios may focus on entire messages. For the purposes of thisdisclosure, the term network message will be used to refer to a networkmessage that may be fragmented into one or more packets unless otherwiseindicated.

This aspect of the network testing system focuses on the measurement ofand visibility into the latency observed in the lab environment. Thereporting period may be subdivided into smaller periodic windows toillustrate trends over time. A standard deviation of measured latenciesmay be measured and reported within each measurement window. Countstracking how many packets fall within each of a set of latency rangesmay be kept over a set of standard-deviation-sized intervals. Latencyboundaries of ranges may be modified for one or more subsequentintervals, based at least in part on the average and standard deviationmeasured in the previous interval. Where these enhanced measurements aretaken during a simulation, they may be presented to a user to illustratehow network latency was affected over time by events within thesimulation.

Average

In certain embodiments, each packet transmitted has a timestamp embeddedin it. When the packet is received, the time of receipt may be comparedagainst the transmit time to calculate a latency. A count of packets anda running total of all latency measurements may be kept over the courseof a single interval. At the end of the measurement interval, an averagelatency value may be calculated by dividing the running total by thecount from that interval, and the count and sum may be reset to zero tobegin the next interval. In some embodiments, a separate counter may bekept to count all incoming packets and may be used to determine theaverage latency value.

Standard Deviation

For a subset of the packets (e.g., one out of every n packets, where nis a tunable parameter), the latency may be calculated as above, and arunning sum of the latency of this subset may be kept. In addition, arunning sum of the square of the latencies measured for this subset maybe calculated. Limiting the calculation to a subset may avoid theproblem of arithmetic overflow when calculating the sum of squares. Atthe end of each interval, the standard deviation over the measuredpackets may be calculated using the “sum of squares minus square ofsums” method, or

$\sigma = \; {{sqrt}( {{{sum}( x^{2} )} - \frac{( {{sum}(x)} )^{2}}{n}} )}$

In certain embodiments, a set of counters may be kept. A first pair ofcounters may represent latencies up to one standard deviation from theaverage, as measured in the previous measurement window. A second pairof counters may represent between one and two standard deviations fromthe average. A third pair of counters may represent two or more standarddeviations from the average. Other arrangements of counters may bevaluable. For example, additional counters may be provided to representfractional standard deviation steps for a more granular view of thedata. In another example, additional counters may be provided torepresent three or four standard deviations away to capture the numberof extreme latency events. In some embodiments, the focus of theanalysis is on high latencies. In these embodiments, one counter maycount all received packets with a latency in the range of zero units oftime to one standard deviation above the average.

The counters may be maintained as follows. For each packet received, oneof the counters is incremented based on the measured latency of thatpacket. At the end of each interval, the counts may be recorded (e.g.,in a memory, database, or log) before the counters are reset. Also atthe end of an interval, the boundaries between counters may be adjustedbased on the new measured average and standard deviation.

In some embodiments, the interval length may be adjusted to adjust thefrequency of measurement. For example, a series of short intervals maybe used initially to calibrate the ongoing measurement and a series oflonger intervals may be used to measure performance over time. Inanother example, long intervals may be used most of the time to reducethe amount of data gathered with short intervals interspersed regularlyor randomly to observe potentially anomalous behavior. In yet anotherexample, the interval length may be adjusted based on an internal orexternal trigger.

In some embodiments, the counters may be implemented within thecapture/offload CLDs 102A. Locating the counters and necessary logicwith CLDs 102A ensures maximal throughput of the statistical processingsystem and maximal precision without the possibility of side effects dueto internal transfer delays between components within the networktesting system.

FIG. 43 illustrates the latency performance of the device orinfrastructure under test as it is presented to a user, according tocertain embodiments of the present disclosure. The chart presentslatency as a function of time. Each column of the chart represents atime slice. Line 900 represents the average latency for messagesreceived in that time slice. Each of blocks 902, 904, 906, and 908represent bands of latencies, e.g., bands bounded by a multiple ofstandard deviations from average. In some embodiments, block 902represents all messages received within the current time slice withlatencies greater than two standard deviations from the average latencyfor the immediately preceding time slice. If no messages are received inthat time slice meeting that criteria, then block 902 will not appearfor that time slice. Similarly, block 904 represents all messagesreceived within the current time slice with latencies greater than onestandard deviation above the average but less than two standarddeviations above the average. Block 906 represents all messages receivedwithin the current time slice with latencies within one standarddeviation of the average. Block 908 represents all messages receivedwithin the current time slice with latencies more than one standarddeviation blow the average but less than two standard deviations belowthe average. The edges of each block center and spread of a standarddeviation curve measured in the immediately preceding time slice.

FIG. 44 is a table of a subset of the raw statistical data from whichthe chart of FIG. 43 is derived, according to certain embodiments of thepresent disclosure. The table includes a timestamp of the first messagereceived within a time slice. The average latency represents the averagelatency for the messages received within that time slice. The next fivecolumns of data indicate the bounds of each of five bands of latencies.These bounds may be described as threshold ranges. The final fivecolumns indicate the number of messages received within each of the fivebands.

FIG. 45 is an example method 920 of determining dynamic latency bucketsaccording to some embodiments of the present disclosure. The method ofFIG. 45 may be performed entirely in capture/offload CLDs 102A as theimplementing logic is sufficiently simple and because delay incalculating latency or new latency threshold values might interfere withthe system's ability to process each received network message within theappropriate time interval. This method will be described in relation tothe five buckets illustrated in FIGS. 43 and 44, though it is notlimited to any particular number of buckets. The initial values of thethreshold ranges may be set to values retrieved from a database ofpreviously captured latencies or may be set arbitrarily. Asynchronous tothis process is a parallel process that is generating and sendingoutbound network messages from the network testing device for whichresponsive network messages are expected.

Process 922 continues for a specified interval of time (e.g., onesecond). In process 922, a responsive network message is received atstep 924 and stamped with a high-resolution clock value indicating atime of receipt. This responsive network message is examined andinformation is extracted that may be used to determine a when acorresponding outbound network message was sent. In some embodiments,the responsive network message includes a timestamp indicating when thecorresponding outbound network message was sent. In other embodiments, aserial number or other unique identifier may be used to lookup atimestamp from a database indicating when the corresponding outboundnetwork message was sent. At step 926, the latency is calculated bysubtracting the sent timestamp of the outbound network message from thereceipt timestamp.

At step 928, the latency is compared against a series of one or morethreshold values to determine which bucket should be incremented. Eachbucket is a counter or tally of the number of packets received with alatency falling within the range for that bucket. In certainembodiments, the threshold values are represented as a max/min pair oflatency values representing the range of values associated with aparticular bucket. The series of buckets forms a non-overlapping, butcontinuous range of latency values. In the example illustrated in FIG.44, in the initial configuration (at time equals zero), the lowestlatency bucket is associated with a range of zero to less than 10microseconds, the second latency bucket is associated with a range often microseconds to less than 100 microseconds, and so forth. In theillustration in FIG. 44, the lowest latency range starts at zero andhighest latency range continues to infinity in order to include allpossible latency values. In some embodiments, the latency ranges may notbe all inclusive and extreme outliers may be ignored. As a final stepwith each received network message, two interval totals are incremented.The first is a total latency value. This total latency value isincremented by the latency of each received packet. The second is a sumof squares value, which is incremented by the square of the latency ofthe received packet, at step 930.

At the end of the time interval, process 932 stores the currentstatistics and adjusts the threshold values to better reflect theobserved variation in latencies. First, the current latency counts andlatency threshold range information is stored at step 934 for laterretrieval by a reporting tool or other analytical software. In someembodiments, the information stored at this step includes all of theinformation in FIG. 44. Next, new threshold latency values arecalculated at step 936.

In some embodiments, step 936 adjusts the threshold latency values tofit a bell curve to the data of the most recently captured data. In thisprocess, the total received message count (maintained independently orcalculated by summing the tallies in each bucket) and the total latencyare used to calculate the average latency, or center of the bell curve.Then, the sum of squares value is used in combination with the averagelatency to determine the value of a latency that is one standarddeviation away from the average. With the average and standard deviationknown, the threshold ranges may be calculated to be: zero to less thantwo standard deviations below the average, two standard deviations toless than one standard deviations below the average, one standarddeviation below to less than one standard deviation above, one standarddeviation above to less than two standard deviations above the average,and two standard deviations above the average to infinity. Finally, thetotal latency and total sum of squares latency values are zeroed at step938.

In embodiments where the threshold latency values do not encompass allpossible latency values, outliers may be completely ignored, or may beused to only calculate the new threshold latency values. In the formercase, step 930 will be skipped for each outlier message so as not toskew the average and standard deviation calculation. In the latter case,a running tally of all received messages is necessary and step 930 willbe performed on all received messages.

Serial Port Access in Multi-Processor System

Serial ports on various processors in system 16 may need to be accessedduring manufacturing and/or system debug phases. In conventionalsingle-processor systems, serial port access to the processor istypically achieved by physically removing the board from the chassis andconnecting a serial cable to an on-board connector. However, this mayhinder debug ability by requiring the board to be removed to attach theconnector, possibly clearing the fault on the board before the processorcan even be accessed. Further, for multi-processor boards of variousembodiments of system 16, the conventional access technique wouldrequire separate cables for each processor. This may cause increasedcomplexity in the manufacturing setup and/or require operatorintervention during the test, each of which may lengthen the test timeand incur additional per board costs. Thus, system 16 incorporates aserial port access system 950 that provides serial access to anyprocessor on any card 54 in system 16 without having to remove any cards54 from chassis 50.

FIG. 46 illustrates an example serial port access system 950 of system16 that provides direct serial access to any processor on any card 54 insystem 16 (e.g., control processors 106 and network processors 105) viathe control processor 106 on any card 54 or via an external serial porton any card 54 (e.g., when control processors are malfunctioning).Serial port access system 950 includes various components of system 16discussed above, as well as additional devices not previously discussed.As shown, serial port access system 950 on card 0 in slot 0 includes acrossbar switch 962 hosted on a CPLD (Complex Programmable Logic Device)123, an external serial port 966 (in this example, an RS-232connection), a backplane MLVDS (Multipoint LVDS) serial connection 952,a management microcontroller 954, an I2C IO expander 956, and abackplane I2C connection 958. Cards 1 and 2 in slots 1 and 2 may includesimilar components.

The crossbar switch 962 on each card 54 may comprise an “any-to-any”switch connected to all serial ports on the respective card 54. As shownin FIG. 46, crossbar switch 962 connects serial ports of controlprocessor 106 (e.g., Intel X86 processor), each network processor 105(e.g., XLR Network processors), external RS-232 connection 966, a sharedbackplane MLVDS connection 952, and management microcontroller 954 toprovide direct serial communications between any of such devices. Inparticular, the serial ports may be set up to connect between any twoattached serial ports through register writes to the CPLD 123. Crossbarswitch 962 may comprise custom logic stored on each CPLD 123.

An MLVDS (Multipoint LVDS) shared bus runs across the multi-bladechassis backplane 56 and allows connectivity to the crossbar switch 962in the CPLD 123 of each other card 54 in the chassis 50. Thus, serialport access system 950 allows access to serial ports on the same blade54 (referred to as intra-blade serial connections), as well as to serialports on other blades 54 in the chassis 50 via the MLVDS shared bus(referred to as inter-blade serial connections).

FIG. 47 illustrates an example method 970 for setting up an intra-bladeserial connection, e.g., when a processor needs to connect to a serialport on the same blade 54. At step 972, a requesting device on aparticular blade 54 sends a command to the control processor 106 forserial access to a target device on the same blade 54. At step 974,control processor 106 uses it's direct register access to CPLD 123containing the crossbar switch 962 to write registers and set up thecorrect connection between the requesting device and target device onblade 54. When the connection is made the two devices act as if theirserial ports are directly connected. This connection will persist untila command is sent to control processor 106 to switch crossbar switch 962to a new serial connection configuration, as indicated at step 976, atwhich point the control processor 106 uses it's direct register accessto CPLD 123 to write registers and set up the new connection between thenew requesting device and new target device (which may or may not be onthe same blade 54).

FIG. 48 illustrates an example method 980 for setting up an inter-bladeconnection between a requesting device on a first blade 54 with a targetdevice on a second blade 54. At step 982, a requesting device on a firstblade 54 sends a command to the local control processor 106 for serialaccess to a target device on a second blade 54. At step 984, the controlprocessor 106 on the first blade 54 sets the local CPLD crossbar switch962 to connect the serial port of the requesting device with the sharedbackplane serial connection 952 on the first blade 54. The sharedbackplane serial connection 952 uses a MLVDS, or Multipoint Low VoltageDifferential Signal, bus to connect to each other blade 54 in the system16. MLVDS is a signaling protocol that allows one MLVDS driver along thenet to send a signal to multiple MLVDS receivers, which allows a singlepin to be used for carrying each of the TX and RX signals (i.e., a totalof two pins are used) and allows inter-blade communication between anyserial ports on any blade 54 in chassis 50. Protocols other than MLVDSwould typically require a separate TX and RX signal for each blade inthe system. Further, MLVDS communications are less noisy than certainother communication protocols, e.g., RS-232.

In addition to setting the registers on the CPLD 123 on the local blade54, control processor 106 sends a message to the local managementmicrocontroller 954 at step 986 to initiate an I2C-based signaling forsetting the CPLD crossbar switch 962 on the second blade as follows. Atstep 988, the management microcontroller 954 uses it's I2C connectivityto the other blades 54 in the system to write to an I2C I/O expander 956on the second blade 54 involved in the serial connection (i.e., theblade housing the target device). For example, the managementmicrocontroller 954 sets 4 bits of data out of the I/O expander 956 onthe second blade 54 that are read by the local CPLD 123. Based on these4 bits of data, CPLD 123 on the second blade 54 sets the local crossbarconfiguration registers to connect the backplane serial MLVDS connection952 on the second blade with the target device on the second blade atstep 990. This creates a direct serial connection between the requestingdevice on the first blade and the target device on the second blade viathe MLVDS serial bus bridging the two blades.

Thus, serial port access system 950 (a) provides each processor insystem 16 direct serial access each other processor in system 16, and(b) provides a user direct serial access to any processor in system 16,either by way of control processor 106 or via external RS-232 serialport 966. If control processor 106 has booted and is functioningproperly, a user can access any processor in system 16 by way of thecontrol processor 106 acting as a control proxy, e.g., according to themethod 970 of FIG. 47 (for intra-blade serial access) or the method 980of FIG. 48 (for intra-blade serial access). Thus, control processor 106can be used as a control proxy to debug other devices in system 16.

Alternatively, a user can access any processor in system 16 via physicalconnection to external RS-232 serial port 966 at the front of chassis50. For example, a user may connect to external RS-232 serial port 966when control processors 106 of system 16 are malfunctioning, not booted,or otherwise inaccessible or inoperative. Serial ports are primitiveperipherals that allow basic access even if EEPROMs or other memorydevices in the system are malfunctioning or inoperative. In addition,CPLD 123 is booted by its own internal flash memory program 960 andaccepts RS-232 signaling/commands, such that crossbar switch 962 in CPLD123 may be booted and operational even when control processors 106and/or other devices of system 16 are malfunctioning, not booted, orotherwise inaccessible or inoperative. As another example, a user mayconnect a debug device or system to external RS-232 serial port 966 forexternal debugging of devices within system 16.

Thus, based on the above, serial port access system 950 includingcrossbar switch 962 allows single point serial access to all processorsin a multi-blade system 16, and thus allows debugging withoutspecialized connections to system 16.

USB Device Initialization

System 16 includes multiple programmable devices 1002 (e.g.,microcontrollers) that must be programmed before each can perform itsassigned task(s). One mechanism for programming a device 1002 is toconnect it to a non-transient programmable memory (e.g., EEPROM orFlash) such that device 1002 will read programming instructions fromthat memory on power-up. This implementation requires a separatenon-transient programmable memory per device 1002, which maysignificantly increase the part count and board complexity. In addition,a software update must be written to each of these non-transientprogrammable memories. This memory update process, often called“flashing” the memory, adds further design complexity and, ifinterrupted, may result in a non-functioning device.

Instead of associating each programmable device 1002 with its ownmemory, some embodiments of the present disclosure provide acommunication channel between control processor 106 and at least somedevices 1002 through which processor 106 can program each device 1002from device images 1004 stored on drive 109. In these embodiments,updating a program for a device 1002 may be performed by updating a fileon drive 109. In some embodiments, a universal serial bus (USB)connection forms the communication channel between control processor 106and programmable devices 1002 through which each device 1002 may beprogrammed.

In an embodiment with one programmable device 1002, that device willautomatically come out of reset and appear on the USB bus ready to beprogrammed. Control processor 106 will scan the USB bus for programmabledevices 1002 and find one ready to be programmed. Once identified,control processor 106 will locate a corresponding image 1004 on drive109 and will transfer the contents of image 1004 to device 1002, e.g.,via a set of sequential memory transfers.

Certain embodiments require additional steps in order identify andprogram specific programmable devices 1002. The programmable devices arenot pre-loaded with instructions or configuration information and eachwill appear identical as it comes out of reset, even though each must beprogrammed with a specific corresponding image 1004 in order to carryout functions assigned to that device within system 16. The USB protocolcannot be used to differentiate devices as it does not guarantee whichorder devices will be discovered or provide any other identifyinginformation about those devices. As a result, control processor 106cannot simply program devices 1002 as they are discovered becausecontrol processor 106 will not be able to identify the specificcorresponding image 1004 associated with that device.

In one embodiment, each programmable device 1004 may be connected to anEEPROM or wired coding system (e.g., DIP switches or hardwired boardtraces encoding a device identifier) to provide minimal instructions oridentification information. However, while this technique may enabledevice-specific programming, it involves initial pre-programming stepsduring the manufacturing process which may add time, complexity, andcost to the manufacturing process. Further, this technique may reducethe flexibility of the design precluding certain types of futuresoftware updates or complicating design reuse.

In some embodiments, system 16 includes a programmable device initiationsystem 1000 that uses one of the programmable devices 1002 (e.g., a USBconnected microcontroller) as a reset master for the other programmabledevices 1002, which allows the slave devices 1002 to be brought out ofreset and uniquely identified by control processor 106 in a staggeredmanner, to ensure that each programmable device 1002 receives the propersoftware image 1004. These embodiments may eliminate the need for anEEPROM associated each USB device discussed above, and may thuseliminate the time and cost of pre-programming each EEPROM.

FIG. 49 illustrates an example USB device initiation system 1000 for usein system 16, according to an example embodiment. As shown, a pluralityof programmable devices 1002, in this case Microcontroller 1,Microcontroller 2, Microcontroller 3, . . . Microcontroller n, areconnected to control processor 106 by USB. Microcontrollers 1-n maycomprise any type of microcontrollers, e.g., Cypress FX2LP EZ-USBmicrocontrollers. Disk drive 109 connected to control processor 106includes a plurality of software images 1004, indicated as Image 1,Image 2, Image 3, . . . Image n that correspond by number to themicrocontrollers they are intended to be loaded onto. Disk drive 109also stores programmable devices initiation logic 1006 (e.g., a softwaremodule) configured to manage the discovery and initiation ofmicrocontrollers 1002, including loading the correct software image 1004onto each microcontroller 1002. Logic 1006 may identify a masterprogrammable devices (e.g., Microcontroller 1 in the example discussedbelow), as well as an order in which the multiple programmable deviceswill be brought up by control processor 106 and a corresponding orderingof images 1004, such that the ordering can be used to match each image1004 with its correct programmable device 1002.

In some embodiments, master programmable device 1002 has outputsconnected to reset lines for each of the slave programmable devices 1002as illustrated in FIG. 50. In other embodiments, master programmabledevice 1002 has fewer outputs connected to a MUX to allow control ofmore slave devices with fewer output pins. In certain embodiments,master programmable device 1002 has one output controlling the resetline of a single other programmable device 1002. That next programmabledevice also has an output connected to the reset line of a thirdprogrammable device 1002. Additional programmable devices may be chainedtogether in this fashion where each programmable device may beprogrammed and then used as a master to bring the next device out ofreset for programming.

FIG. 50 illustrates an example method 1020 for managing the discoveryand initiation of microcontrollers 1002 using the programmable deviceinitiation system 1000 of FIG. 49, according to an example embodiment.One of the programmable devices, in this example Microcontroller 1, ispre-selected as the master programmable device prior to system boot up,e.g., during manufacturing. At step 1022, system 16 begins to boot up.The pre-selected master programmable device, Microcontroller 1, comesout of reset as the system powers up (and before control processor 106completes its boot process). Due to the operation of the pull-downcircuits on the other programmable devices (indicated in FIG. 49 bypull-down resistors R_(PD)), Microcontrollers 2-n, are held in reset atleast until Microcontroller 1 has been programmed. At step 1024, controlprocessor 106 (e.g., and Intel x86 processor running an operating systemloaded from drive 109) boots up and performs a USB discovery process onthe USB bus, and sees only Microcontroller 1. In response, at step 1026,control processor 106, having knowledge that Microcontroller 1 is themaster USB device (as defined in logic 1006), determines from logic 1006that Image 1 corresponds with Microcontroller 1, and thus programsMicrocontroller 1 with Image 1 from drive 109. Once Microcontroller 1 isprogrammed, control processor 106 can access it via the USB connectionand control the resets to the other USB devices. Thus, control processor106 can then cycle through the USB devices one at a time, releasing themfrom reset, detecting them on the USB bus, and then programming thecorrect image on each device, as follows.

At step 1028, control processor 106 releases the next programmabledevice from reset using reset signaling shown in FIG. 49-1 by drivingthe output high that is connected to the reset pin on the nextprogrammable device to be programmed, e.g., Microcontroller 2. At step1030, control processor 106 detects this next device on the USB bus asready to be programmed, determines using logic 1006 the image 1004 ondrive 109 corresponding to that programmable device, and programs thatimage 1004 onto the programmable device. Using this method controlprocessor 106 can cycle through the programmable devices(Microcontrollers 2-n) one by one, in the order specified in logic 1006,to ensure that each device is enumerated and programmed for correctsystem operation. Once control processor 106 determines that allprogrammable devices have come up, the method may end, as indicated atstep 1032.

CLD Programming Via USB Interface and JTAG Bus

Programming via USB Interface

Past designs have used different methods to program CLDs and have causeddesign and update issues:

Programming from local flash/EEPROM: This method programs the CLDsimmediately on boot so the parts are ready very quickly, however it alsorequires individual flash/EEPROM parts at each CLD. Also, CLD designfiles have become quite large (e.g., greater than 16 MB), and that filesize is increasing software update time by requiring as much as fiveminutes per CLD to overwrite each flash/EEPROM memory.

Programming via software through CPLDs: This is another standard methodto use the Fast Parallel programming method for the CLDs. In thisapproach, software installed on a CPLD from internal flash memoryinitiates the programming during each boot process. Connectivity to theCPLD from the control processor can be an issue with limited optionsavailable. To use a PCI connection between control processor 106 and aCLD to be programmed, the CPLD must implement PCI cores, which consumesvaluable logic blocks and requires a licensing fee. Other communicationoptions require the use of specialized integrated circuits. Moreover,this approach requires complex parallel bus routing to connect the CPLDto each CLD to be programmed. Long multi-drop parallel busses need to becorrectly routed with minimal stubs and the lengths need to becontrolled to maintain signal integrity on the bus. Some embodimentshave 5 FPGA's placed across an 11″x18″ printed computer board (PCB)resulting in long traces.

To enable fast, flexible programming of CLDs, an arrangement ofcomponents is utilized to provide software-based programming of CLDscontrolled by control processor 106. In certain embodiments, one or moremicrocontrollers are provided to interface with the programming lines ofCLDs (e.g., the Fast Parallel Programming bus on an FPGA). Those one ormore microcontrollers are also connected to control processor 106 via ahigh speed serial bus (e.g., USB, IEEE 1394, THUNDERBOLT). The smallsize of the microcontroller combined with the simplified trace routingenabled by the serial bus allowed direct, high speed programming accesswithout the need for long parallel bus lines. Furthermore, adding one ormore additional microcontrollers could be accomplished with minimalnegative impact to the board layout (due to minimal part size and wiringrequirements) while allowing for further simplification of parallel busrouting.

FIG. 51 illustrates the serial bus based CLD programming system 1050according to certain embodiments of the present disclosure. System 1050includes control processor 106 coupled to drive 109, andmicrocontrollers 1052, and CLDs 102. Drive 109 includes CLD access logic1054 (i.e., software to be executed on microcontrollers 1052) and CLDprogramming images 1056. Control processor 106 is coupled tomicrocontrollers 1052 via a high-speed serial bus (e.g., USB, IEEE 1394,THUNDERBOLT). Microcontrollers 1052 are coupled to CLDs 102 viaindividual control signals and a shared parallel data bus.

In certain embodiments, two microcontrollers (e.g., Cypress FX2 USBMicrocontrollers) are provided. One is positioned near two CLDs 102 onone side of the board, and the other is positioned on the opposite sideof the board near the other three CLDs 102. This placement allows forshort parallel bus connections to each CLD to help ensure signalintegrity on those busses.

FIG. 52 illustrates an example programming process 1060 according tocertain embodiments of the present disclosure. At step 1062, system 16powers up and control processor 106 performs its boot process to load anoperating system and relevant software modules. During this step,microcontrollers 1052 will power up and will signal availability forprogramming to control processor 106 via one or more serial connections(e.g., USB connections). At step 1064, control processor 106 locateseach microcontroller 1052 and transfers CLD access logic images 1054from disk 109 to each microcontroller. In some embodiments, an identicalCLD access logic image 1054 is loaded on each microcontroller. Incertain embodiments, each microcontroller 1052 has identifyinginformation or is wired in a master/slave configuration (e.g., in asimilar configuration as shown in FIG. 49) such that control processor106 may load a specific CLD access logic image 1054 on eachmicrocontroller 1052.

At step 1066, control processor 106 communicates with eachmicrocontroller 1052 via CLD access logic to place the CLDs inprogramming mode. Microcontroller 1052 may perform this operation bydriving one or more individual control signals to initiate a programmingmode in one or more CLD 102. In some embodiments, microcontroller 1052may program multiple CLD 102 simultaneously (e.g., with an identicalimage) by initiating a programming mode on each prior to transmitting aprogramming image. In some embodiments, microcontroller 1052 may programCLD 102 devices individually.

At step 1068, control processor 106 locates CLD image 1056 correspondingto the next CLD to program. Control processor 106 may locate thecorresponding image file based on information hard-coded on one or moredevices. In some embodiments, microcontrollers 1052 may have one or morepins hard-coded (e.g., tied high grounded by a pull-down resistor) toallow specific identification by control processor 106. In theseembodiments, that identification information may be sufficient to allowcontrol processor 106 to control a specific CLD 102 by driving apredetermined individual control signal line. In other embodiments,microcontrollers 1052 are programmed identically while CLDs 102 may havehard-coded pins to allow identification by the correspondingmicrocontroller 1052. In these embodiments, CLD access logic 1054 willinclude logic to control each CLD 102 individually in order to read thehard-coded pins and thereby identify that device by type (e.g.,capture/offload CLD or L2/L3 CLD) or specifically (e.g., a specific CLDwithin system 16).

Once the corresponding CLD image has been identified, control processor106 transfers the contents of that image (e.g., in appropriately sizedsub-units) to microcontroller 1054 via the serial connection.Microcontroller 1054, via an individual control signal, initiates aprogramming mode on the CLD being programmed and loads image 1056 intothe CLD via the shared parallel data bus.

At step 1070, control processor 106 determines whether another CLD 102should be programmed and returns to step 1066 until all have beenprogrammed.

The transfer speed of the serial bus (e.g., USB) is sufficiently fast totransfer even large (e.g., 16 MB) image files in a matter of seconds toeach CLD. This programming arrangement also simplifies updates wherereplacing CLD image files 1056 on drive 109 will result in a CLDprogramming change after a restart. No complicated flashing (andverification) process is required.

Programming Via JTAG Bus

Any time flash memories or EEPROMs are updated through software there isa risk of corruption that may result in one or more non-functionaldevices. The present disclosure provides a reliable path to both programon-board devices such as CLD's as well as on-board memories (e.g.,EEPROMs and flash memory). The present disclosure also provides areliable path to recover from a corrupted image in most devices withoutrendering a board into a non-functional state (a.k.a., “bricking” aboard). The present disclosure additionally provides a path fordebugging individual devices.

In-system programming of all programmable devices on board is criticalfor field support and software upgrades. Past products did not have agood method for in system programming some devices and caused fieldreturns when an update was needed or to recover from a corrupted device.The present disclosure provides a method to both update all chips as apart of the software upgrade process and to be able to recover from acorrupted image in an on-board memory device (e.g., EEPROM or flash).

In addition to image update and field support, the present disclosurealso provides more convenient access to each CLD for in-system debug.Previous designs required boards be removed and cables attached to runthe debug tools. The present disclosure provides in-place, in-systemdebug capability. This capability allows debugging of a condition thatmay be cleared by removing the board from the system.

FIG. 53 illustrates debug system 1080, according to certain embodimentsof the present disclosure. Debug system 1080 includes JTAG code image1088 (e.g. stored in drive 109), microcontroller 1082, control processor106, JTAG chains 1092 and 1094, and demultiplexers 1084. Controlprocessor 106 may load JTAG code image 1088 on microcontroller 1082(e.g., over a USB connection) as part of the system boot sequence. Insome embodiments, microcontroller 1082 is a CYPRESS microcontroller).JTAG code image 1088 provides software for implementing the JTAG busprotocol under interactive control by control processor 106.Demultiplexer 1084 enables segmentation of the JTAG bus into shortsegment 1092 and long segment including 1092 and 1094. In someembodiments, a multiplexer (controlled by the same bus select line) maybe inserted between the JTAG chain input and both FPGA 102 and MAC 330to create two independent JTAG busses. In these embodiments,demultiplexer 1086 is no longer necessary and the last FPGA 102 beforethat demultiplexer may be connected directly to demultiplexer 1084. Incertain embodiments, JTAG chain input is a set of electrical connectionsincluding test mode select (TMS), test clock (TCK), and a directlyconnected test data in (TDI) connection. Each device in the chain has adirect connection between its test data out (TDO) pin and the nextdevice's TDI pin, except where the final TDO connects to thedemultiplexer.

To allow for both programming and CLD debug, the JTAG chain has beensubdivided into two sections. The first section includes each CLD andthe second section includes all other JTAG compatible devices in system16. This division enables convenient access to and automatic recognitionof ALTERA devices by certain ALTERA-supplied JTAG debug tools.

In certain embodiments, short chain 1092 provides JTAG access to the 5FPGA's and 3 CPLD's on the board. This mode may be used to program theCPLD's on the board, to program the Flash devices attached to twoCPLD's, and to run the ALTERA-supplied debug tools. The ALTERA tools arerun through a software JTAG server interface. ALTERA tools running on aremote workstation may connect via a network connection to controlprocessor 106 and access the JTAG controller. Control processor 106 mayinclude a modified version of the standard LINUX URJTAG (Universal JTAG)program to enable CPLD and flash programming. Through that tool, controlprocessor 106 may program the CPLD's, and through the programmed CPLD's,the tool can access each attached flash memory not directly connected tothe JTAG bus. The flash memories may contain boot code for one or morenetwork processors. Use of the JTAG bus to program these flash memoriesenables programming of the boot code without the processor running.Previous designs had to be pre-programmed and had the risk of “bricking”a system if a re-flash was interrupted. Recovery from such aninterruption required a return of the entire board for lab repair.System 1080 allows the boot code to be programmed regardless of thestate of the network processor allowing for in-field update andrecovery.

When attached to the full chain (e.g., 1092 and 1094) themicrocontroller has access to all the devices on the JTAG bus. The fullchain may be used to program the Serial Flash containing the boot codefor the networking switch 110 on the board. To program networking switch110, the JTAG software on control processor 106 may control the pins ofnetworking switch 110 to write out a new flash image indirectly.

Branding Removable or Replaceable Components

As with many systems, drive 109 is a standard size and has a standardinterface making it mechanically and electrically interchangeable withcommodity hardware. However, not all drives have satisfactoryperformance and reliability characteristics. In particular, while asolid state device may provide sufficiently low access times andsufficiently high write throughput to maintain certain applications, aphysically and electrically compatible 5,400 RPM magnetic drive mightnot. In some cases, high-volume purchasers of drives may purchasecustomized devices with manufacture supplied features for ensuring thatonly authorized drives are used within a system. To prevent users fromoperating system 16 with an unauthorized drive, control processor 106may read certain information from drive 109 to verify that the drive isidentified as an authorized drive.

FIG. 69 illustrates a drive branding solution, according to certainembodiments of the present disclosure. In some embodiments, drive 109 isa persistent storage device such as a solid state drive (SSD) incommunication with control processor 106 via a SATA interface. Drive 109may include manufacture supplied read only memory 1350 including uniqueserial number 1355. Manufacturers provide unique serial numbers onstorage devices to track manufacturing quality, product distribution,and purchase/warranty information. Read only memory 1350 may bepermanently set in a write-once memory, e.g., in a controller circuit orread-only memory (ROM) device.

In some embodiments, drive 109 may be partitioned into two logicalunits, hidden partition 1351, including branding information 1356, anddata partition 1352. In some embodiments, hidden partition 1350 may be adrive partition formatted, for example, in a non-standard format. Incertain embodiments, hidden partition 1351 may be a standard drivepartition formatted as a simple, standard file system (e.g., FAT). Insome embodiments, branding information 1356 may be a raw data written toa specific block on hidden partition 1351. In some embodiments, brandinginformation 1356 may data written to a file on hidden partition 1350.

Data partition 1352 may be a standard drive partition formatted as astandard file system (e.g., FAT, ext2, NTFS) and may contain operatingsystem and application software, CLD images, packet capture data, andother instructions and data required by system 16.

FIG. 70 illustrates branding and verification processes, according tocertain embodiments of the present disclosure.

Branding process 1360 may include the following steps performed by aprocessor such as processor 106 on a second drive 109. At step 1361,software executing on processor 106 may read the drive serial numberfrom read only memory 1350. At step 1362, that software may partitionthe drive into a hidden partition 1251 and a data partition 1352. Atstep 1363, the software may format hidden partition 1251. In someembodiments, step 1363 may be skipped if formatting is not required(e.g., where branding information 1356 is written as raw data to aspecific block of partition 1351). At step 1364, the drive serial numberis combined with secret information using a one-way function such as thejhash function or a cryptographic hash to obtain branding information1356. At step 1365, branding information 1356 is written to hiddenpartition 1351. At this point, the drive will be recognized asauthorized by system 16 and data partition 1352 may be formatted andloaded with an image of system 16.

Verification process 1370 may include the following steps performed byCPU 134. At step 1371, CPU 134 powers up and loads the basic inputoutput system (BIOS) instructions stored in SPI EEPROM. At step 1372,CPU 134 accesses drive 109 and loads branding information 1356 and driveserial number 1355. At step 1373, CPU 134 verifies branding information1356. In some embodiments, CPU 134 may apply a public key (which pairswith the private key used in step 1364) to decrypt branding information1356. If the decrypted value matches serial number 1355, the drive maybe recognized as authorized. In other embodiments, CPU 134 may combineserial number 1355 with the same secret used in step 1364 and in thesame manner. If the result is the same as branding information 1356, thedrive may be recognized as authorized.

If the drive is authorized, CPU 134 may begin to boot the operatingsystem from partition 1352 at step 1374. If the drive is not authorized,CPU 134 may report an error at step 1375 and terminate the boot process.The error report may be lighting a light emitting diode (LED) on thecontrol panel of system 16.

In some embodiments, verification process 1370 may be performed bysoftware executed by the operating system as part of the operatingsystem initialization process.

Physical Design Aspects and Heat Dissipation

As discussed above, network testing system 16 may comprise one or moreboards or cards 54 arranged in slots 52 defined by a chassis 50. FIG. 54illustrates one example embodiment of network testing system 16 thatincludes a chassis 50 having three slots 52 configured to receive threecards 54. Each card 54 may have any number and types of externalphysical interfaces. In the illustrated example, each card 54 has aremovable disk drive assembly 1300 that houses a disk drive 109; one ormore ports 1102 for connection to a test system 18 for management oftest system 18, one or more ports 1104 (e.g., including RS-232 port 996)for connection to controller 106 for managing aspects of card 54, a port1106, e.g., a USB port for inserting a removable drive for performingsoftware upgrades, software backup and restore, etc., for debugging card54 (e.g., by connecting a keyboard and/or mouse to communicate with thecard 54), or for any other purpose; and a number of ports 1100corresponding to test interfaces 101. Each card 54 may also include apower button and any suitable handles, latches, locks, etc., forinserting, removing, and/or locking card 54 in chassis 50.

Heat dissipation presents significant challenges in some embodiments ofsystem 16. For example, CLDs 102, processors 105 and 106, and managementswitch 110 may generate significant amounts of heat that need to betransferred away from system 16, e.g., out through openings in chassis50. In some embodiments, limited free space and/or limited airflowwithin chassis 50 present a particular challenge. Further, in someembodiments of a multi-slot chassis 50, different slots 52 receivedifferent amounts of air flow from one or more fans, and/or the physicaldimensions of individual slots (e.g., the amount of free space above thecard 54 in each respective slot 52) may differ from each other, theamount of volume and speed of air flow. Further, in some embodiments,the fan or fans within the chassis 50 tend to move air diagonally acrossthe cards 54 rather than directly from side-to-side or front-to-back.Further, heat-generated by one or more components on a card 54 maytransfer heat to other heat-generating components on the card 54 (e.g.,by convection, or by conduction through the printed circuit board), thusfurther heating or resisting the cooling of such other heat-generatingcomponents on the card 54. Thus, each card 54 may include a heatdissipation system 1150 that incorporates a number of heat transfersolutions, including one or more fans, heat sinks, baffles or other airflow guide structures, and/or other heat transfer systems or structures.

FIGS. 55A-59B illustrate various views of an example arrangement ofdevices on a card 54 including a heat dissipation system 1150, atvarious stages of assembly, according to an embodiment that correspondswith the embodiment shown in FIGS. 14A and 14B. In particular, FIGS. 55Aand 55B show a three-dimensional view and a top view, respectively, ofthe example card 54 with heat-management components and removable diskdrive assembly 1300 removed, in order to view the arrangement of variouscomponents of card 54. FIGS. 56A and 56B show a three-dimensional viewand a top view, respectively, of card 54 with heat sinks and removabledisk drive assembly 1300 installed. FIGS. 57A and 57B show athree-dimensional view and a top view, respectively, of card 54 with atwo-part air baffle 1200 installed, in which a first part 1202 of theair baffle 1200 is shown as a transparent member in order to view anunderlying second part 1204 of air baffle 1200. FIGS. 58A and 58B show athree-dimensional view and a top view, respectively, of card 54 with thefirst part 1202 of the air baffle 1200 removed in view the underlyingsecond part 1204 of air baffle 1200. Finally, FIGS. 59A and 59B show athree-dimensional view and a top view, respectively, of card 54 with thefirst part 1202 of the air baffle 1200 installed over the second part1204 and shown as a solid member.

Turning first to FIGS. 55A and 55B, card 54 includes a printed circuitboard 380 that houses a pair of capture and offload CLDs 102 a-1 and 102a-2 and associated DDR3 SDRAM memory modules (DIMMs) 103A-1 and 103A-2,a pair of routing CLDs 102 b-1 and 120 b-2 and associated QDR SRAMs 103b-1 and 103 b-2, a traffic generation CLD 102C, a pair of networkprocessors 105-1 and 105-2 and associated DDR2 SDRAM DIMMs 344-1 and344-2, a control processor 106 and associated DDR3 SDRAM DIMMs 332, amanagement switch 110, four test interfaces 101, a backplane connector328, a notch or bay 388 that locates a drive connector 386 for receivinga disk drive assembly 1300 that houses a disk drive 109, and variousother components (e.g., including components shown in FIGS. 14A and14B). As shown, DIMMS 103A-1, 103A-2, 344-1, 344-2, and 332 may bealigned in the same direction, e.g., in order to facilitate air flowfrom one or more fans across card 54 in that direction, e.g., in adirection from side-to-side across card 54.

Turning next to FIGS. 56A and 56B, a number of heat sinks may beinstalled on or near significant heat-generating devices of card 54. Asshown, card 54 includes a dual-body heat sink 1120 to remove heat fromfirst network processor 105-1, a heat sink 1122 to remove heat fromsecond network processor 105-2, a heat sink 1124 to remove heat fromcontrol processor 106, a number of heat sinks 1126 to remove heat fromeach CLD 102 a-1, 102 a-2, 102 b-1, 102 b-2, and 102 c, and a heat sink1128 to remove heat from management switch 110. Each heat sink may haveany suitable shape and configuration suitable for removing heat from thecorresponding heat-generating devices. As shown, each heat sink mayinclude fins, pegs, or other members extending generally perpendicularto the plane of the card 54 for directing air flow from one or more fansacross the card 54. Thus, the fins of the various heat sinks may bealigned in one general direction, the same alignment direction as DIMMS103A-1, 103A-2, 344-1, 344-2, and 332, in order to facilitate air flowin a general direction across card 54 through the heat sinks and DIMMS.Some heat sinks may include an array of fins in which each individualfins extends in one direction (the direction of air flow), and with gapsbetween fins that run in a perpendicular direction, which gaps maycreate turbulence that increases convective heat transfer from the finsto the forced air flow.

As discussed below in greater detail, dual-body heat sink 1120 forremoving heat from first network processor 105-1 includes a first heatsink portion 1130 arranged above the network processor 105 and a secondheat sink portion 1132 physically removed from network processor 105 butconnected to the first heat sink portion 1130 by a heat pipe 1134. Heatis transferred from the first heat sink portion 1130 to the second heatsink portion 1132 (i.e., away from network processor 105) via the heatpipe. As shown in FIGS. 56A and 56B, the second heat sink portion 1132may be arranged laterally between two sets of DIMMs 103A-2 and 344-1,and longitudinally in line with another set of DIMMs 344-2 in thegeneral direction of air flow. Details of dual-body heat sink 1120 arediscussed in more detail below with reference to FIGS. 60-62.

FIGS. 57A-59B show various views of a two-part air baffle 1200 installedover a portion of card 54 to manage air flow across card 54. Two-partair baffle 1200 includes a first part 1202 and an underlying second part1204. In FIGS. 57A and 57B, first part 1202 of air baffle 1200 is shownas a transparent member in order to view the underlying second part1204. In FIGS. 58A and 58B, first part 1202 of air baffle 1200 isremoved for a better view of the underlying second part 1204. Finally,in FIGS. 59A and 59B, first part 1202 is shown as a solid memberinstalled over the second part 1204.

As shown in FIGS. 57A-59B, air baffle 1200 may include variousstructures and surfaces for guiding or facilitating air flow across card54 as desired. For example, first part 1202 of air baffle 1200 mayinclude a thin, generally planar sheet portion 1206 arranged abovecomponents on card 54 and extending parallel to the plane of the printedcircuit board, and a number of guide walls 1214 extending downwardly andperpendicular to the planar sheet portion 1206. Similarly, second part1204 may include a thin, generally planar sheet portion 1216 arrangedabove components on card 54 and extending parallel to the plane of theprinted circuit board, and a number of guide walls 1212 extendingdownwardly and perpendicular to the planar sheet portion 1206. Guidewalls 1212 and 1214 are configured to influence the direction and volumeof air flow across various areas and components of card 54, e.g., topromote and distribute air flow through the channels defined betweenheat sink fins and DIMMs on card 54.

In addition, first part 1202 of air baffle 1200 may include angled flapsor “wings” 1208 and 1210 configured to direct air flow above air baffle1200 downwardly into and through the fins of heat sinks 1120 and 1122,respectively, to promote conductive heat transfer away from such heatsinks. As discussed below with reference to FIG. 65, wings 1208 and 1210may create a low pressure area that influences air flow downwardly intothe respective heat sinks

Details of air baffle 1200 is discussed in more detail below withreference to FIGS. 63-65.

Dual-Body Heat Sink

As discussed above, heat dissipation system 1150 of card 54 may includea dual-body heat sink 1120 that functions in cooperation with air baffle1200 to dissipate heat from a network processor 105 (e.g., a NetlogicXLR 732 1.4 GHz processor).

FIGS. 60-62 illustrate details of an example dual-body heat sink 1120,according to one embodiment. In particular, FIG. 60 shows athree-dimensional isometric view, FIG. 61 shows a top view, and FIG. 62shows a bottom view of heat sink 1120. As shown, a first heat sink body1130 and a second heat sink body 1132 may each include an array of fins1220 or other members for encouraging convention from bodies 1130 and1132 to an air flow.

First heat sink body 1130 is connected to the spaced-apart second heatsink body 1132 by a heat pipe 1134. As shown in FIG. 62, heat sink 1120may include two heat pipes: a first heat pipe 1134 that connects firstheat sink body 1130 with second heat sink body 1132, and a second heatpipe 1152 located within the perimeter of first heat sink body 1130. Athermal interface area 1160 in which network processor 105-1 physicallyinterfaces with heat sink body 1130 is indicated in FIG. 62. Both heatpipes 1134 and 1152 extend through the thermal interface area 1160 tofacilitate the movement of heat from processor 105-1 to heat sink bodies1130 and 1132 via the thermal interface area 1160. Heat pipe 1134 movesheat to the remotely-located heat sink body 1132, which is cooled by anair flow across heat sink body 1132, which causing further heat flowfrom heat sink body 1130 to heat sink body 1132. Two heat sink bodiesare used so that memory (DIMMs 344-1) for processor 105-1 can be placedclose to processor 105-1. The cooling provided by the dual-body designmay provide increased or maximized processing performance of processor105-1, as compared with certain single-body heat sink designs.

As shown, both heat pipes 1134 and 1152 interface with processor 105-1via thermal interface area 1160. The co-planarity of this interface maybe critical to adequate contact. Thus, the interface may be milled to avery tight tolerance. Further, in some embodiments, a phase changethermal material or other thermally-conductive material may be providedat the interface to ensure that heat sink body 1130 is bonded at themolecular level with processor 105-1. This material may ensuresextremely high thermal connectivity between processor 105-1 and heatsink body 1130.

In this embodiment, each heat pipe is generally U-shaped, and isreceived in rectangular cross-section channels 1162 milled in heat sinkbodies 1130 and 1132, except for the portion of pipe 1134 extendingbetween first and second heat sink bodies 1130 and 1132. Each channel1162 may be sized such that a bottom surface of each heat pipe 1134 and1152 is substantially flush with bottom surfaces of heat sink bodies1130 and 1132. Thus, heat pipes 1134 and 1152 are essentially embeddedin heat sink bodies 1130 and 1132. Heat pipes 1134 and 1152 may haverounded edges. Thus, when heat pipes 1134 and 1152 are installed inchannels 1162, gaps are formed between the walls of channels 1162 andthe outer surfaces of heat pipes 1134 and 1152. Left empty, such gapswould reduce the surface area contact between the heat pipes and theheat sink bodies, as well as the contact between the heat pipes/heatsink and processor 105-1 at thermal interface area 1160, which mayreduce the performance of processor 105-1. Thus, such gaps between thewalls of channels 1162 and the outer surfaces of heat pipes 1134 and1152 may be filled with a thermally conductive solder or other thermallyconductive material to promote heat transfer between heat pipes 1134 and1152 and heat sink bodies 1130 and 1132, and all bottom surfaces maythen be machined flat, to provide a planar surface with a tighttolerance.

Heat sink bodies 1130 and 1132 and heat pipes 1134 and 1152 may beformed from any suitable thermally-conductive materials. For example,heat sink bodies 1130 and 1132 may be formed from copper, and heat pipes1134 and 1152 may comprise copper heat pipes embedded in copper heatsink bodies 1130 and 1132.

Fins 1220 on bodies 1130 and 1132 may be designed to provide a desiredor maximum amount of cooling for the given air flow and air pressure forthe worst case slot 52 of the chassis 50. The thickness and spacing offins 1220 may be important to the performance of heat sink 1120.Mounting of heat sink 1120 to card 54 may also be important. Forexample, thermal performance may be degraded if the pressure exerted onheat sink 1120 is not maintained at a specified value or within aspecified range. In one embodiment, an optimal pressure may be derivedby testing, and a four post spring-based system may be designed andimplemented to attach heat sink 1120 to the PCB 380.

In some embodiments, fans in chassis 50 create a generally diagonal airflow though the chassis 50. Due to this diagonal airflow, as well as therelatively small cross section of cards 54 and “pre-heating” ofprocessors caused by heat from adjacent processors, a special air baffle1200 may be provided to work in conjunction with heat sink 1120 (andother aspects of heat dissipation system 1150), as discussed above. Airbaffle 1200 has unique features with respect to cooling of electronic,and assists the cooling of other components of card 54, as discussedabove with reference to FIGS. 57A-59B and below with reference to FIGS.63-65.

Air Baffle

In some embodiments, management switch 110 generates large amounts ofheat. For example, management switch 110 may generate more heat than anyother device on card 54. Thus, aspects of heat dissipation system 1150,including the location of management switch 110 relative to othercomponents of card 54, the design of heat sink 1128 coupled tomanagement switch 110, and the design of air baffle 1200, may bedesigned to provide sufficient cooling of management switch 110 forreliable performance of switch 110 and other components of card 54.

As shown in FIGS. 55A and 55B, in the desired direction of air flowacross card 54, management switch 110 is aligned with network processor105-1. Due to the large amount of heat generated by switch 110, it maybe disadvantageous to dissipate heat from management switch 110 into theair flow that subsequently flows across and through heat sink 1130 abovenetwork processor 105-1. That is, delivering a significant portion ofthe heat from switch 100 through the heat sink intended to cool networkprocessor 105-1 may inhibit the cooling of network processor 105-1.Thus, heat sink 1128 may be configured to transfer heat from managementswitch 110 laterally, out of alignment with network processor 105-1 (inthe desired direction of air flow). Thus, as shown in FIGS. 56A and 56B,heat sink 1128 may include a first conductive portion 1136 positionedover and thermally coupled to management switch 110, and a second finnedportion 1138 laterally removed from management switch 110 in order toconductively transfer heat laterally away from management switch 110 andthen from the fins of finned portion 1138 to the forced air flow byconvection. In this example configuration, finned portion 1138 isaligned (in the air flow direction) with DIMMs 344-1 rather than withnetwork processor 105-1. Because DIMMs typically generate substantiallyless heat than network processors, DIMMs 344-1 may be better suited thannetwork processor 105-1 to receive the heated airflow from switch 110.

Further, as shown in FIGS. 57A-57B and 58A-58B, air baffle 1200 isconfigured to direct and increase the volume of air flow across heatsink 1128. For example, angled wing 1210 directs air flow downwardlythrough heat sink 1122, which then flows through heat sink 1128.Further, an angled guide wall 1212 of the second part 1204 of air baffle1200 essentially funnels the air flow to heat sink 1128, thus providingan increased air flow mass and/or speed across heat sink 1128.

FIGS. 63-65 provide views of example air baffle 1200 removed from card54, to show various details of air baffle 1200, according to oneembodiment. FIG. 63 shows a three-dimensional view from above air baffle1200, in which first part 1202 of air baffle 1200, also referred to as“shell” 1202, is shown as a transparent member in order to view theunderlying second part 1204, also referred to as “air deflector” 1204.FIG. 64A shows a three-dimensional exploded view from below of shell1202 and air deflector 1204. FIG. 64A shows a three-dimensionalassembled view from below of air deflector 1204 received within shell1202. Finally, FIG. 64A shows a side view of assembled air baffle 1200,illustrating the directions of air flow promoted by air baffle 1200, inparticular angled wings 1208 and 1210, according to one embodiment.

In one embodiment, shell 1202 is a sheet metal shell, and air deflector1204 serves as a multi-vaned air deflector that creates specificchannels for air to flow. The parts are assembled as shown in FIGS. 64Aand 64B. As discussed above, the sheet metal shell 1202 may includeslanted wing like structures 1210 and 1208, which act as low pressuregenerators to direct air flow downwardly as shown in FIG. 65. Similar toan aircraft wing, an angle of attack with respect to the plane of thesheet metal (1206 in FIG. 65) may be set for each wing 1210 and 1208,indicated as θ₁ and θ₂, respectively. The angles θ₁ and θ₂ may beselected to provide desired air flow performance, and may be the same ordifferent angles. In some embodiments, one or both of θ₁ and θ₂ arebetween 20 and 70 degrees. In particular embodiments, one or both of θ₁and θ₂ are between 30 and 60 degrees. In certain embodiments, one orboth of θ₁ and θ₂ are between 40 and 50 degrees.

Each wing 1210 and 1208 creates a low pressure area, which deflects aportion of the air flow above the sheet metal plane 1206 downwardly intothe air baffle 1200. This mechanism captures air flow that wouldnormally move above the heat sink fins and redirects this air flowthrough the heat sink fins. The redirected airflow may be directed tolower parts of the heat sinks located within the air baffle (i.e., belowthe sheet metal plane 1206), thus providing improved coolingperformance. An indication of air flow paths provided by air baffle 1200is provided in FIG. 63.

Further, as discussed above, air baffle 1200 may include guide vanes1214 and 1212 extending perpendicular from planar sheets 1206 and 1214of shell 1202 and air deflector 1204 (i.e., downwardly toward PCB 380).As discussed above, fans may tend to generate a diagonal air flow acrosscard 54. On a general level, guide vanes 1214 and 1212 may direct thisair flow across card 54 in a perpendicular or orthogonal to the sides ofcard 54, rather than diagonally across card 54, which may promoteincreased heat dissipation. On a more focused level, as shown in FIGS.63 and 64B, particular guide vanes 1212 of air deflector 1204 may beangled with respect to the perpendicular side-to-side direction of airflow, which may create areas of increased air flow volume and/or speed,e.g., for increased cooling of management switch 110, as discussedabove. In one embodiments, vanes 1214 and 1212 are implemented as aLexan structure. Thus, to summarize, in some embodiments, vanes 1214 and1212 linearize the diagonal air flow supplied by high speed fans inchassis 50. The vanes cause the air to flow through/over the heat sinkswithin and downstream of air baffle 1200, which may provide the airspeed and pressure necessary for proper operation of such heat sinks.Further, vanes 1214 and 1212 may be designed to substantially preventpre-heated air from flowing through critical areas that may require orbenefit from lower-temperature air for desired cooling of such areas,e.g., to substantially prevent air heated by management switch 110 byway of heat sink 1128 from subsequently flowing across downstream heatsink part 1130 arranged above network processor 105-1.

Drive Carrier

As discussed above, in some embodiments, disk drive 109 is a solid statedrive that can be interchanged or completely removed from card 54, e.g.,for interchangeability security and ease of managing multiple projects,for example. Disk drive 109 may be provided in a drive assembly 1300shown in FIGS. 56A and 56B. Drive assembly 1300 includes a drive carriersupport 1340 that is secured to card 54 and a drive carrier 1302 that isremoveably received in the drive carrier support 1340. Drive carrier1302 houses solid state disk drive 109, which is utilized by controlprocessor 106 for various functions, as discussed above. With referenceto FIGS. 55A-55B and 56A-56B, drive carrier support 1340 may be receivedin notch 388 formed in PCB 380 and secured to PCB 380. When drivecarrier 1302 is fully inserted in drive carrier support 1340,connections on one end of disk drive 109 connect with drive connector386 shown in FIGS. 55A and 55B, thus providing connection between drive109 and control processor 106 (and/or other processors or devices ofcard 54).

FIGS. 66-68B illustrate various aspects of drive assembly 1300,according to one example embodiment. FIG. 66 shows an assembled drivecarrier 1302, according to the example embodiment. Drive carrier 1302comprises a disk housing 1304 for housing disk drive 109. In oneembodiment, disk housing 1304 may substantially surround disk drive 109,but provide an opening at one end 1307 of the housing 1304 to allowexternal access to an electrical connector 1305 of disk drive, which isconfigured to connect with electrical connector 386 on PCB 380 in orderto provide data communications between disk drive 109 and components ofcard 54.

Lateral sides 1308 of disk housing 1304 are configured to be slidablyreceived in guide channels of drive carrier support 1340, shown in FIGS.68A and 68B. Disk housing 1304 may also include end flanges 1312 thatinclude a groove 1310 or other protrusion or detent for engaging withspring tabs 1345 at the back portion of drive carrier support 1340,shown in FIGS. 68A and 68B. Disk housing 1304 may also include a lightedlabel 1314 and a handle 1306 for installing and removing drive carrier1302. Handle 1306 may comprise a D-shaped finger pull or any othersuitable handle.

FIG. 67 shows an exploded view of drive carrier 1302, according to theexample embodiment. As shown, drive carrier 1302 includes disk drive 109sandwiched between an upper housing 1322 and a lower housing 1320. Alight pipe or light guide 1324 is also housed between upper housing 1322and lower housing 1320, which delivers light to a front label 1314, anda faceplate 1330 having an opening is assembled over label carrier 1314.Any suitable light source may be used for lighting label 1314, e.g., apair of multicolored LEDs positioned on each lateral side of the drivecarrier 1302 on the PCB 380. A top label 1328 may be attached to the topof drive carrier 1302.

FIGS. 68A and 68B show details of drive carrier support 1340, accordingto an example embodiment. Drive carrier support 1340 may include a body1342 having guide channels 1344 on opposing lateral sides for slidablyreceiving lateral sides 1308 of disk housing 1304. Drive carrier support1340 may also include flanges 1346 for securing drive carrier support1340 to PCB 380, and spring tabs 1345 having protrusions configured toengage with grooves 1310 formed in the end flanges 1312 of drive carrier1302 (shown in FIG. 66). The location of spring tabs 1345 and grooves1310 may provide precise positioning of drive carrier 1302 in thedirection of insertion, which may ensure proper connection with driveconnector 386. The interaction between spring tabs 1345 and grooves 1310provides a latching mechanism that provides a spring-based latchingforce that secures drive carrier 1302 in drive carrier support 1340, butwhich can be overcome by a user pulling handle 1306 to remove drivecarrier 1302 out of drive carrier support 1340. Drive carrier support1340 may thus serve to align the drive carrier 1302, provide a smoothslide during insertion, provide depth control, and a latching mechanismto secure the drive carrier 1302.

The components of drive carrier 1302 and drive carrier support 1340 maybe formed from any suitable materials. In some embodiments, drivecarrier 1302 may be formed from materials that provide desired weight,conductivity, and/or EMI shielding, e.g., aluminum.

Drive carrier support 1340 may be formed from any suitable materials. Insome embodiments, drive carrier support 1340 may be formed frommaterials that provide low insertion force (e.g., low friction force).For example, drive carrier support 1340 may be formed frompolyoxymethylene, acetal, polyacetal, or polyformaldehyde to provide aself-lubricating surface, rigidity, stability, and machinability.

In some embodiments, drive assembly 1300 and/or card 54 includes a drivestatus detection system for automatic detection of the removal orinsertion of drive carrier 1302 from drive carrier support 1340. Forexample, the drive status detection may include an electrical microswitch configured to detect the presence or absence of the drive carrier1302 (or communicative connection/disconnection of drive 109 from card54). Other embodiments include software for detecting the presence orabsence of the drive carrier 1302 (or communicativeconnection/disconnection of drive 109 from card 54). Such software mayperiodically check an ID register on the drive 109 to verify that thedrive carrier 1302 is still installed. If the drive is not found, thesoftware may automatically issue a board reset. A special BIOS functionmay be provided that periodically or continuously checks for a drive 109if a drive is not found. Once the drive carrier 1302 is installed andthe BIOS detects the drive 109, the card 54 will boot normally.

For the purposes of this disclosure, the term exemplary means exampleonly. Although the disclosed embodiments are described in detail in thepresent disclosure, it should be understood that various changes,substitutions and alterations can be made to the embodiments withoutdeparting from their spirit and scope.

What is claimed is:
 1. A method of initializing programmable devices ona shared bus, comprising: on power up, loading control instructions on aprocessor from an attached tangible, non-transient computer-readablemedium; automatically allowing a first programmable device on a commonbus to exit its reset state and enter a read-to-program state as thecomputer system powers up; automatically holding a second programmabledevice on the common bus in its reset state; querying the common bus bythe processor to identify the first programmable device; copying thecontents of a program image file by the processor from thecomputer-readable medium across the common bus to the first programmabledevice; and once the first programmable device has been programmed withthe contents of the program image file, signaling the first programmabledevice to release the reset hold on the second programmable device. 2.The method of claim 1, wherein the common bus is a universal serial bus(USB).
 3. The method of claim 1, wherein the first and secondprogrammable devices are USB microcontrollers.
 4. The method of claim 1,wherein a pull-down resistor automatically holds down the reset pin ofthe second programmable device and wherein the first programmable devicehas a pin connected to the reset pin of the second programmable device.5. The method of claim 4, wherein a pull-down resistor automaticallyholds down the reset pin of the third programmable device on the commonbus and wherein the first programmable device has a pin connected to thereset pin of the third programmable device.
 6. The method of claim 4,wherein a pull-down resistor automatically holds down the reset pin ofthe third programmable device on the common bus and wherein the secondprogrammable device has a pin connected to the reset pin of the thirdprogrammable device.
 7. A tangible, non-transitory computer-readablemedia comprising instructions that, when executed on a processor, enablethe processor to perform the steps comprising: query a common bus by aprocessor to identify a first programmable device connected to thecommon bus; copy the contents of a program image file by the processorfrom the computer-readable medium across the common bus to the firstprogrammable device; and once the first programmable device has beenprogrammed with the contents of the program image file, signal the firstprogrammable device to release a reset hold on a second programmabledevice.
 8. The method of claim 7, wherein the common bus is a universalserial bus (USB).
 9. The method of claim 7, wherein the first and secondprogrammable devices are USB microcontrollers.
 10. The method of claim7, wherein a pull-down resistor automatically holds down the reset pinof the second programmable device and wherein the first programmabledevice has a pin connected to the reset pin of the second programmabledevice.
 11. The method of claim 10, wherein a pull-down resistorautomatically holds down the reset pin of the third programmable deviceon the common bus and wherein the first programmable device has a pinconnected to the reset pin of the third programmable device.
 12. Themethod of claim 10, wherein a pull-down resistor automatically holdsdown the reset pin of the third programmable device on the common busand wherein the second programmable device has a pin connected to thereset pin of the third programmable device.
 13. A computing system,comprising: a processor attached to a tangible, non-transientcomputer-readable medium containing instructions and a first and asecond program image file; and a common bus connected to the processorand a first and a second programmable device; wherein: the firstprogrammable device is configured to exit its reset mode automaticallyon power up; the second programmable device is configured to remain inits reset mode until signaled by the first programmable device; and theinstructions, when executed by the processor, instruct the processor toperform the steps comprising: query the common bus to identify the firstprogrammable device; copy the contents of a program image file by theprocessor from the computer-readable medium across the common bus to thefirst programmable device; and once the first programmable device hasbeen programmed with the contents of the program image file, signal thefirst programmable device to release a reset hold on a secondprogrammable device.
 14. The system of claim 13, wherein the common busis a universal serial bus (USB).
 15. The system of claim 13, wherein thefirst and second programmable devices are USB microcontrollers.
 16. Thesystem of claim 13, wherein a pull-down resistor automatically holdsdown the reset pin of the second programmable device and wherein thefirst programmable device has a pin connected to the reset pin of thesecond programmable device.
 17. The system of claim 16, wherein apull-down resistor automatically holds down the reset pin of the thirdprogrammable device on the common bus and wherein the first programmabledevice has a pin connected to the reset pin of the third programmabledevice.
 18. The system of claim 16, wherein a pull-down resistorautomatically holds down the reset pin of the third programmable deviceon the common bus and wherein the second programmable device has a pinconnected to the reset pin of the third programmable device.